task, and facilitates model debugging to achieve the original task step by
step. Note that this is not equivalent to using agent’s solution to each sub-
task as input to the next sub-task, but rather continuously realizing new sub-
tasks based on past tasks. That is, this is not a splitting of tasks, but an
overlay of the sub-tasks to realize the original task.

Design Steps. The design of a simulation model often begins with the
implementation of simple rules, which are then gradually expanded to encompass
more complex scenarios. This approach allows for the verification of the
model’s accuracy in a simplified context before moving on to more intricate
rules. Drawing from our experience with the number-guessing game, we propose
the following step-by-step process for adding sub-tasks to the simulation
model:

  1. 1.

Base Modeling: The base model should be as simple as possible, focusing on the
core functionality of the agents. A key issue in base modeling is setting the
essential parameters for the simulation. This includes the configuration of
the LLM and initialization of variables. In the number-guessing game, the
agent configuration could include the choice of the temperature and max_token
parameters. The initialization could include the range of numbers to guess
from and the method for setting the target number. In addition to parameter
setting, advanced agent modeling methods (e.g., reasoning, planning, and
personalization) are sometimes necessary for the agents to yield reasonable
outputs. This could involve defining the agents’ goals, preferences, and
behaviors, which can guide their interactions with other agents and the
environment. For example, agents may need to reason or plan before presenting
the answer, especially when handling mathematical or logical tasks.

  2. 2.

Data Calibration: After the base model has been created, it can be calibrated
with real-world data. Data collected from real-world behaviors (e.g., human
volunteers’) can serve as ground truth. The calibration could involve the
adjustment of model parameters or personalization prompts to better reflect
the data or the use of few-shot examples to guide the model’s behavior. An
example of such practice will be demonstrated in the case study in Section 7.

  3. 3.

Sub-task Addition: Once the model is calibrated, additional sub-tasks can be
added. A typical sub-task is testing different personas for the agents. Other
sub-tasks include creating dialogues between agents, defining more complex
rules for interactions, and introducing new tasks for the agents to perform.
Besides, we can scale up the number of agents to make the model results more
general.

###  5.4 Implementation and Simulation

With the designed model, we can implement it and run the simulation. In
particular, debugging is a crucial process, allowing us to identify and
correct any issues that may arise. The steps we propose for implementing the
model and running the simulation are carried out in a bottom-up fashion and
listed as follows.

  1. 1.

Implementation: The coding of the model may involve both programming language
and natural language, where prompt engineering techniques can be used to
refine the natural language prompts. The code is suggested to be written in a
modular and reusable manner, with clear comments and documentation. This will
facilitate debugging and make the model easier to understand and modify.

  2. 2.

Single-Agent Debugging: Firstly, we need to ensure that the behavior and
decision-making patterns of individual agents are reasonable and align with
actual real-world situations. Prompts that are described inaccurately or
ambiguously can lead agents to misunderstand and exhibit biased behavior. This
can result in unexpected outcomes when multiple agents interact, making it
difficult to pinpoint the causes of such behaviors. To address this, we can
employ single-agent debugging, which focuses solely on understanding what
information an individual agent receives and how it might react. By adjusting
the information provided to the agent (which, in actual simulations, is
generated through interactions with other agents and/or the environment), we
can observe the agent’s reactions under different circumstances and modify the
prompts accordingly to align the agent’s behavior with reality. Since
debugging a single agent involves fewer inputs and outputs compared to multi-
agent interactions, this approach can effectively reduce both financial and
computational costs. An example of such practice will be demonstrated in the
case study in Section 6.

  3. 3.

Sub-task Analysis: For each sub-task, we systematically observe the agents’
actions under various situations with the same prompt configuration. By
comparing the agents’ behaviors with existing theories and ground truth, we
can assess the reasonableness of the model construction and the prompt. This
process is akin to an ablation study [32], where each component of the model
is individually tested and its contribution to the overall model performance
is evaluated.

  4. 4.

Sub-task Combination: If the addition of a sub-task improves the model’s
ability to reflect the phenomena being studied, then the sub-task is deemed
effective. If the addition of a sub-task does not improve the model, we need
to verify its feasibility in an individual model before attempting to re-add
it to the combined task.

  5. 5.

Holistic Task Simulation: Finally, we perform debugging in the context of a
holistic task simulation. This allows us to identify any issues that may arise
when all components of the model are working together. It is suggested that a
save/load function is implemented for debugging, so we do not need to simulate
from scratch when a bug is identified.

For instance, when studying emergency evacuation (to be presented in Section
6), we could start by simulating a scenario where there is no persona or
communication between agents. We could then add personas for agents, and
finally introduce communication between agents. This step-by-step approach
allows us to gradually increase the complexity of the simulation, ensuring
that each component is functioning correctly before moving on to the next.

###  5.5 Validation

We have elucidated the methods of applying SABM to simulation modeling.
However, the creation of a model is only one part of the process. The ultimate
goal of simulation is not the model itself, but its ability to represent the
reality of the problem under study. If the model fails to do so, even the most
optimal solution derived from it may not be applicable to the real-world
problem. Therefore, it is crucial to have methods in place to validate the
simulation and ensure the reliability of the model.

Traditional computer simulation often employs sensitivity analysis as a means
of testing model reliability. This method measures the sensitivity of the
model’s state changes in response to alterations in system parameters and
conditions [129]. Sensitivity analysis allows for a comprehensive
understanding of the model’s behavior under different conditions, thereby
providing a robust measure of its reliability.

In the context of SABM, we need to pay additional attention to the testing of
the prompt. ABM approaches may utilize pre-trained language models (e.g.,
[175]), but language is seldom part of the input parameters. This is where
SABM diverges. In SABM, language, in the form of a prompt, plays the role of
fundamental parameters and serves as an essential model component.
Consequently, it has a more significant impact on the model’s results.

We need to scrutinize how and to what extent the prompt influences the
behavior and results of the simulation. This is crucial to prevent the
simulation from becoming overly dependent on a specific prompt, which could
lead to a loss of reproducibility. It is essential to ensure that the model
remains robust and reliable across a range of prompts, thereby enhancing its
real-world relevance and applicability. Next, we will delve into the methods
and strategies for prompt testing in SABM.

Prompt Alteration. In order to verify the stability of the model, we propose
three strategies for altering the prompt, each reflecting a different role
that the prompt plays in a simulation:

  1. 1.

Paraphrasing: This strategy involves changing the wording or order of the
prompt without altering its meaning, to test whether the agent’s behavior and
the system’s behavior are sensitive to the prompt. For instance, if the
model’s output change significantly after altering you may to you might, it
would suggest that the model is overly sensitive to the prompt.

  2. 2.

Varying Elements: This strategy involves changing some of the simulation’s
settings and observing the resulting changes in the model. While the original
objective of the simulation remains unchanged, the model’s performance is
affected by altering the meaning of statements, examples of prompts, and by
increasing or decreasing the information or constraints provided to the agent.
For example, changing you can to you cannot, adding few-shot examples,
providing or withholding formulas and values for certain variables in the
prompt to the agents, etc. It is expected that such changes will alter the
model’s performance to some degree, but the difference in performance before
and after the modification should be explainable and consistent with existing
theories or empirical data.

  3. 3.

Varying Objectives: This strategy involves changing the objectives of the
simulation to test the reliability of the prompt. Changes to the simulation
objectives may have the most significant impact on the model results, but such
changes need to be interpretable. For instance, we might change the nature of
a game from a win-win to a zero-sum game in a game of chance. In such a case,
there is a higher likelihood that the agent’s behavior will change. If the
behavior persists, or if the change in behavior cannot be rationally explained
by theory, it may indicate that the model does not adequately understand the
task description.

Table 3: Observations of variations in simulation results. Extent of Variation |  Definition
---|---
Low |  No significant change
Medium |  Quantitative change with significance (in terms of output value)
High |  Qualitative change with significance (in terms of agent behavior pattern)
Table 4: Examples of prompt alteration for the number-guessing game at different levels. Extent of Variation |  Modified Prompt |  Original Prompt |  Results and Analysis
---|---|---|---
Low |  Paraphrasing: Pick an integer from 1 to 100. |  The number will be an integer ranging from 1 to 100. |  No significant difference in the results. (interpretable)
Medium |  Varying Elements (in the prompt for domain knowledge): You cannot use binary search. |  You can use binary search to optimize your guess. |  We observe that when the guesser is instructed not to use binary search, the number of attempts needed to reach the target number increases significantly compared to the case when it is allowed to use it (on average from 5.18 to 6.45 over 100 runs, $p<0.001$ using the Mann-Whitney U test), confirming significant quantitative changes in the results. (interpretable)
High |  Varying Objectives: The number will be an even integer ranging from 1 to 100. |  The number will be an integer ranging from 1 to 100. |  The search space of the model produces a change, but the task can still be handled using a binary search algorithm. The specific guesses in the results change (no odd numbers occur), but the trend and algorithm remain the same. (interpretable)
Table 5: Summary of case studies.

Case number |  1 |  2 |  3
---|---|---|---
Scenario |  Emergency evacuation |  Plea bargaining |  Firm pricing competition
Discipline |  Behavioral science |  Criminology |  Economics
Number of agents |  100 – 400 |  1 |  2
Model construction steps |  Preliminary design |  ✓ |  ✓ |
Base modeling |  ✓ |  ✓ |  ✓
Data calibration |  |  ✓ |
Sub-task addition |  ✓ |  ✓ |  ✓
Single-agent debugging |  ✓ |  |
Sub-task analysis |  ✓ |  ✓ |  ✓
Sub-task combination |  ✓ |  ✓ |  ✓
Holistic task simulation |  ✓ |  |  ✓
Validation |  ✓ |  |
Implementation methods |  Common sense |  ✓ |  ✓ |
Domain knowledge |  |  |  ✓
Memory |  ✓ |  |  ✓
Learning |  ✓ |  ✓ |
Reasoning |  |  ✓ |
Planning |  ✓ |  |  ✓
Personalization |  ✓ |  ✓ |  ✓
Conversation |  ✓ |  |  ✓
Model interpretation |  ✓ |  |
Instance components |  Agent: heterogeneity |  Heterogeneous agents |  Heterogeneous agents |  Heterogeneous agents
|  |  |
Agent: alignment |  Common sense |  Common sense |  Domain knowledge
|  |  |
Environment |  Agent behavior-based variation |  None |  Agent behavior-based variation
|  |  |
Interactions |  Local communication |  None |  One-on-one
|  |  |
System setup: temporal structure |  Discrete time |  None |  Discrete time
|  |  |
System setup: spatial structure |  Regular grids |  None |  None

By systematically varying the prompt and observing the resulting changes in
the model, we can gain a deeper understanding of the model’s behavior and its
sensitivity to different prompts. We can also categorize the observed changes
in the results into three types, indicating the magnitude of the changes, and
thus determine whether the model is stable across multiple scenarios. The
classification can be seen in Table 3.

Indeed, the stability of a model is not solely determined by the magnitude of
the observed changes when varying the prompt, but more importantly, by whether
the results align with our expectations. When we paraphrase the original text,
we do not anticipate significant changes in the results (i.e., there should be
a low level of variation in the observations of model outcomes). If the
results are consistent with our expectations or show limited, interpretable
variations, it suggests that the model is stable. On the other hand, if the
model’s results change in ways we do not anticipate or appear random, it
signals instability in the model. This indicates sensitivity to specific
vocabulary or phrasing in the prompt, necessitating further adjustments and
testing to ensure reproducibility. In the case of varying elements or
objectives in tasks, we expect changes in the model’s results. However, these
changes should be explainable based on the modifications made to the prompt
and should align with existing theories or empirical data. If the changes are
inexplicable or inconsistent with our expectations, it may indicate that the
model is overly sensitive to specific aspects of the prompt.

Absolutely, it is crucial to underscore that the observation of changes in
results should typically be based on statistical outcomes. Given that LLMs
inherently produce outputs with a degree of randomness, the agent behaviors
they simulate also embody a certain level of diversity and uncertainty.
Therefore, the statistical results derived from multiple simulations can serve
as a basis for judgment, helping to mitigate any bias that might be introduced
by such randomness. This approach aligns with the principles of Monte Carlo
simulations [76], which rely on repeated random sampling to obtain numerical
results. By running the simulation multiple times and averaging the results,
we can obtain a more accurate and reliable estimate of the model’s behavior
under different prompts. This method can effectively smooth out the randomness
inherent in individual simulation runs, providing a more robust measure of the
model’s sensitivity to changes in the prompt.

In the number-guessing game, we can validate the reliability of the model
through a series of modifications on the prompt. We are able to create
variations of the original prompt at low, medium, and high levels, and observe
the resulting changes in the model’s behavior. Table 4 shows the settings and
results of the modifications.

In summary, these strategies of altering prompts enable us to understand how
specific segments of the prompt influence the behavior and results of our
model. This is crucial for explaining and validating prompt settings, thereby
enhancing the reliability and real-world relevance of our model. By
systematically testing the model’s sensitivity to different prompts, we can
ensure its robustness and applicability across a range of scenarios,
contributing to the overall validity of the SABM approach. Moreover, the
validation of these models cannot be decoupled from qualitative comparisons
with existing theories and real-world phenomena. A strong qualitative
resemblance between a simulation and an actual system serves as indirect
evidence of the simulation’s fidelity [25]. In order to claim that our
simulations accurately reflect real-world systems, we need supporting evidence
from real-world data or theory. This evidence should demonstrate that our
simulations capture the essential characteristics of these systems. Existing
qualitative and quantitative studies need to be taken into account to keep our
models and studies in reality. In certain scenarios, expert validation methods
are also used for model verification, which should also be taken into
consideration [131].

Sensitivity Analysis on Parameters. Sensitivity analysis plays a crucial role
in understanding the impact and validation of various parameters on the
model’s output. This has been extensively discussed in the context of computer
simulations [36, 80, 129]. In the context of SABM, the same principles apply.
While some SABM simulations may solely rely on text prompts without involving
any quantitative variables, others may incorporate parameters in the model. In
such cases, traditional sensitivity analysis methods can be applied to these
parameters in addition to the prompt alteration strategies.

And it is important to note that other than the quantitative variables, a key
aspect to consider when using the GPT model as the core of the agent is the
effect of the temperature parameter on the simulation results. The temperature
parameter in GPT models controls the randomness of the model’s responses, with
higher values leading to more diverse outputs. Therefore, it is essential to
analyze the performance of the simulation results for different temperature
parameters. This will help determine whether the stochasticity and diversity
introduced by the language model significantly affect the simulation.

As per the methodology proposed in this section, we apply SABM to three case
studies. We emphasize that our goal here is to demonstrate the methodology of
SABM rather than extensive experiments and thorough analysis. Nonetheless, we
consider that extending these case studies might lead to more interesting
results and deeper understanding in their respective fields. Table 5
summarizes these case studies, showing the optional steps of model
construction, the implementation methods used, and the categorization of their
components (as per Figure 8). In particular, the first case study is a
complete SABM instance, and the other two delve into agent modeling and
interaction modeling, respectively.

##  6 Case Study 1: Emergency Evacuation

This comprehensive case study simulates the complex patterns of evacuation in
emergency situations. Given the significance of evacuation research for public
safety and risk management, as well as the challenges in obtaining data from
real-world scenarios [33], there have been many simulation studies that
explored the evacuation contexts [88, 156, 179, 180]. In this case study, we
model 100 – 400 heterogeneous agents, each representing an evacuee, and
investigate how SABM can simulate complex behaviors through methods, including
common sense reasoning, memorization, learning, planning, and conversation. To
design the model, we take [156] as the base, a study on the simulation of
emergency evacuation using ABM, and incrementally replace its components with
SABM.

Figure 13: Design of emergency evacuation. Dashed lines represent ABM methods
only used in Sub-task 1.

###  6.1 Task Definition

We simulate scenarios with varying population densities to examine the
dynamics of emergency evacuation. We primarily observe how variations in
individual characteristics, such as physical strength, mental state, and
location-related factors, affect the agents’ decision-making and movement
along with their response to the actions of their peers. We are particularly
interested in whether the presence of congestion causes the agents to abandon
their originally planned escape paths in search of alternatives that are less
congested, despite these alternatives potentially not being the closest exits.

Figure 13 displays the conceptual design of the case study, which will be
detailed in the rest of this section. We set the simulation environment as a
grid environment composed of $33\times 33$ cells, where the evacuation is
simulated by the movement of agents. We use $(i,j)$ to denote the coordinates
of a cell in the grid, where $i$ and $j$ represent row and column, numbered
from top to bottom and left to right, respectively, and both starting from 0.
We have a preliminary design phase where the agents are first tested on their
ability to understand text-based grid scenarios, which serves as the basis for
selecting the LLM to use in the simulation. Subsequently, in the simulation
design, the agents choose the target exit and determine the direction of
movement based on their common sense. In each round, the agents can move from
one cell to another or remain stationary.

At the start of each round, the following situational description is provided
to each agent:

Evacuation scenario Because of the earthquake, you need to escape from the
room where you are as fast as possible. The room has a size of 33 * 33. There
are 3 exits in the room. The exits are located at the left, bottom, and right
of the room. To escape from the room, you need to consider the following two
aspects: exit proximity and people count. The exit proximity is the distance
between you and the nearest exit. The people count is the number of people you
can see. The distance to the nearest exit is {distance_to_nearest_exit}. There
are {number_of_people} people in your visible range.

In each round, an agent’s state is first calculated based on the level of
local congestion at its current position and its abilities to cope with
congestion, including competitive ability and tolerance ability. When an agent
has been overtaken a certain number of times, its state is set to disabled and
it leaves the simulation. Following this, the agent’s movement unfolds in four
stages. In Stage 1, the agents are asked about their feelings regarding the
situation to judge whether they are panicking in the current scenario. In
Stage 2, based on the information available within their field of view,
including the distance to exits and the number of people within the view, the
agents assess the exits. In Stage 3, they decide their respective target exit.
In Stage 4, they determine the direction of movement according to the planed
path towards the target exit in Stage 3. Moreover, we consider conversation as
an option in Stage 3, allowing them to share information with other agents and
to use the information as one of the factors for deciding the target exit.
Specific settings regarding field of view, local congestion, and direction of
movement are detailed in the model design.

###  6.2 Preliminary Design

The objective of the preliminary design is to test whether the LLM can
understand text-based grid scenarios, including whether it can grasp that the
situational task of the evacuation is to reach an exit within the grid as fast
as possible, whether it can accurately understand coordinates information, and
whether the route planning is consistent and rational. To test the LLM’s
capability, we employ two setups to determine the model to be used in the
simulation. There are two model_types to choose from: gpt-3.5-turbo-0301
(referred to as GPT-3.5 for the rest of this section) and gpt-4-0314 (referred
to as GPT-4 for the rest of this section). max_tokens is set to 512 and
temperature is set to 0.0 †††Although setting temperature to zero may result
in limited diversity of behaviors under exactly the same setting, in this
procedurally generated, interactively dynamic environment, we seldom encounter
exactly the same outcome. Meanwhile, in a physically situated setting (e.g., a
grid), the LLM used in this case study still has restricted capabilities on
scene understanding, and increasing the temperature may introduce diversity as
well as unwanted randomness at the same time [94].. The following text about
the grid scenario is provided for testing:

Text-based grid scenario Imagine you are in a room laid out as a 10 * 10 grid.
You are currently at coordinates {initial_position}. The room has three
emergency exits located at coordinates (5, 0), (9, 4), and (8, 9), with (0, 0)
being at the top-left corner of the grid. Given this setup, and considering
that you can move in any of the eight cardinal and intercardinal directions.
It is possible to move diagonally, e.g. from (1, 1) to (2, 2) is one move to
the upper right, and is faster than (1, 1)-$>$(1, 2)-$>$(2, 2). You need to
determine the safest and fastest route to evacuate the room. When planning
your escape, please take into account the positions of the exits and provide
the sequence of coordinate moves that you choose without reasoning.

By testing GPT-3.5 and GPT-4 with two initial positions, (1, 4) and (2, 7), we
show their choice of exit and route in Figure 14. It can be observed that
GPT-4 is capable of accurately understanding the map and planning the shortest
route to the nearest exit, whereas GPT-3.5 fails to find the nearest exit or
the shortest escape route, and exhibits a high level of uncertainty in its
decision. This indicates that GPT-3.5 cannot accurately comprehend text-based
grid scenarios and is incapable of rationally planning escape routes.
Therefore, we will use GPT-4 in the model design.

Figure 14: Output samples of preliminary design.

###  6.3 Model Design

(a) Red and blue points are agents. Yellowish cells indicate the field of view
of the red agent.

(b) Grid with obstacles depicted in slate blue blocks.

Figure 15: Grid environment. White, gray, and pink cells indicate empty space,
walls, and exits, respectively.

####  6.3.1 Grid Environment

The environment is modeled as a grid comprising $33\times 33$ cells, with all
sides bounded by walls and three exits located centrally along the left,
bottom, and right sides. As depicted in Figure 15(a), the right-side exit
consists of a single cell, thereby being narrower than the other two exits,
each extending across three cells. This design intends to introduce a
bottleneck to closely examine if the agents can modify their evacuation
strategies in response to varying congestion levels. As depicted in Figure
15(b), we optionally add obstacles to the scenario, allowing for a comparative
analysis of environments with and without obstacles. Each cell is designated
for single occupancy and may either be empty, occupied by an obstacle, or
occupied by an agent. In case two agents aim to enter the same cell
simultaneously, a random selection process determines which agent will
actually enter the cell.

####  6.3.2 Basic Agent Setup

We outline the components that constitute an agent’s behavior, which include
the agent’s direction of movement, field of view, and local congestion.

Movement. At each round, an agent may move to one of the eight surrounding
cells in the directions: up, down, left, right, upper-left, lower-left, upper-
right, lower-right, or it may remain stationary in its current cell. To ensure
that the agents comprehend the possibility of diagonal movement within the
simulation, we provide an additional (few-shot) instruction to facilitate
learning of these movement patterns.

Instruction for movement You need to escape to the exit as fast as possible.
The room has a size of 33 * 33. We use (i, j) to denote the position, smaller
i means top and bigger i means bottom; smaller j means left and bigger j means
right. Position (1, 1) is at the top left of the room. It is possible to move
diagonally, e.g., from (1, 1) to (2, 2) is one move to the lower right, and is
faster than (1, 1)-$>$(1, 2)-$>$(2, 2).

Field of View. An agent’s field of view is limited to a specific region ahead
of it. As illustrated in Figure 15(a), the diagram showcases the field of view
for the agent in red at position (16, 23) directed towards the exit below.
Within this limited area, an agent gathers local information such as the count
of people and obstacles (e.g., from the view field at (16, 23) looking towards
the bottom exit, there are two other agents), informing subsequent actions.

Local Congestion. In recognition of potential injuries in crowded real-world
situations, the model accounts for overtaking events in congested spaces
during evacuation. The agents are attributed with characteristics that
describe their ability to handle crowded conditions. These characteristics
include competitive ability, tolerance ability, tolerance limit, congestion
degree, and overtaken counts. Essentially, an agent’s competitive ability is
assessed against those of other agents in the adjacent eight cells. By
accumulating the differences in competitive ability, we have the congestion
degree at the agent’s present location. If the congestion degree surpasses the
agent’s tolerance ability, the agent is regarded as overtaken and sustains an
injury, transitions from a normal to a critical state, and halts movement for
one round. Accumulating injuries and overtaken count beyond the tolerance
limit leads an agent to a disabled state, at which point they are removed from
the simulation, indicative of an unsuccessful evacuation. For competitive
ability, tolerance ability, and tolerance limit, we randomly sample a number
from a normal distribution where $\sigma=1$ and $\mu$ is determined by the
agent’s physical strength specified in its persona. For physically strong
agents, $\mu$ is 3, 18, and 26 for the three attributes, respectively. For
physically weak agents, $\mu$ is 2, 16, and 23 for the three attributes,
respectively. The choice of these mean values roughly follows the parameter
settings in [156], and we use normal distributions instead of fixed values for
more variation.

In addition to the above setup, we assume:  (1) The initial positions of the
agents are randomly distributed with uniform probability.  (2) The agents are
presumed to know the coordinates of all exits.

####  6.3.3 Personalization

Our model incorporates heterogeneity among agents using personalization, which
affects their ability to adapt changes in their surroundings and ultimately
influences their capacity for evacuation route planning. For instance, the
agents may become more prone to panic due to changes in the environment under
the influence of their mentally weak persona. The personalization takes into
account attributes of physical and mental aspects, hence including four
personas: physically strong and mentally strong, physically strong but
mentally weak, physically weak but mentally strong, and physically weak and
mentally weak. These personas, used as prompts, affect the agents’ behavioral
characteristics, as given below.

Physically strong and mentally strong persona You are positive and full of
energy, and you are strong and fit.

Physically strong but mentally weak persona You are positive and full of
energy, but you are not a strong person.

Physically weak but mentally strong persona You are negative and afraid of
difficulties, but you are strong and fit.

Physically weak and mentally weak persona You are negative and afraid of
difficulties, and you are not a strong person.

####  6.3.4 Planning

As previously mentioned, there are four stages to decide the movement of an
agent in each round. We employ planning for modeling the movement.

Subjective Feelings. In Stage 1, we model the subjective influencing factors
that affect the agents’ panic levels, which serve as a psychological basis for
their subsequent decision. Due to the difficulty in describing subjective
influencing factors, in the ABM setup in [156], parameters are used to signify
whether the agents are uniformly experiencing panic. In the SABM setup, we
gauge the agents’ feelings towards the current situation to assess their panic
levels with the following prompt.

Stage 1: Subjective feelings {persona}. Please tell me your feelings about the
situation around you in one sentence showing if you are panicking or not.

Assessment and Choice of Target Exits. In the ABM setup, the agents determine
their target exit by using the number of people within their field of view and
the distance to each exit as inputs to an analytical model. In the SABM setup,
the agents apply common sense and planning to evaluate how the given
information about the exits influences their escape. Drawing on the subjective
feelings identified in Stage 1, the assessment of these factors also reflects
individual physical and mental conditions, rather than relying solely on
quantitative measures of distance and congestion levels. For instance, an
agent with superior physical strength may opt for an exit that is further away
but less crowded. As a result, in Stage 2, the agents combine their subjective
feelings with other information, including exit proximity and congestion, to
assess the exits, as shown in the following prompt.

Stage 2: Assessment of target exits {persona}. Now you feel: “{panic_level}”.
Here shows you the distances to different exits and the number of people you
can see towards those exits: Exit {exit_id}: {distance_to_exit} away,
{number_of_agents_around} people around. Please tell me briefly how will you
evaluate the two aspects of each exit based on your personal mental and
physical characteristics in one sentence. Please give 3 sentences for each
exit (around 15 words).

In Stage 3, using the outcomes of the assessment in Stage 2, the agents decide
the target exit.

Stage 3: Choice of target exit {persona}. Now you feel: “{panic_level}”. There
are 3 exits in this room. Based on the current situation, your personal
feelings on each exit are: {assessment}. Please tell me which exit you would
like to choose to escape, and you always want to escape as fast as possible.
Please use the exit id to indicate your choice. For example, if you want to
choose exit left, you can say ’left’. Only output one word of text to indicate
your choice. You can choose from [’bottom’, ’left’, ’right’]. Give your answer
without any additional text.

Decision on Movement. Based on the planning in Stages 2 and 3, the agents
decide the action, i.e., the direction of movement.

Stage 4: Decision on movement Select your move from these possible options
(you can move in diagonal or horizontal directions, options with obstacles or
other people are excluded and not in the path, and option codes are in random
order): {movement_list}
Please tell me your best choice to escape as fast as possible with one single
code without any additional texts. You can choose from {valid_directions}.

####  6.3.5 Memory

Each GPT call is an individual request without previous dialog histories,
while agents should have the sense of its previous decisions to
comprehensively determine their current choices. To mimic the human-like
memorization functionality, we provide agents with summarized historical
information in the prompts to ensure consistency in their evacuation choices,
meanwhile reduce the costs on redundant information. In Stage 3, when
selecting their target exits, they will receive the historical information
about their previously chosen exits. In Stage 4, when determining their
movement directions, they can also obtain their location in the previous
round.

Historical information Target exit history in Stage 3: … your personal
feelings on each exit are: {assessment}. Here are the previous decisions you
made for the target exit from the beginning: {target_exit_history}. This means
most recently you were heading to exit target_exit. Please keep these in mind
when you make your decision.

Please tell me which exit you would like to choose to escape … Position
history in Stage 4: You were at {last_position} last time. To escape from the
room, you have chosen the exit at {exit_position} and you are at
{cur_position}, so the exit is on your {direction(agent_position,
exit_position)}. Select your move from these possible options …

####  6.3.6 Conversation

As an option in this case study, the agents can exchange information,
simulating the communication that would occur between individuals in real
evacuation, including dialogue and potential collaboration. In each round, the
probability that an agent shares information is set to 20%, and the order of
speaking is randomly determined. Conversation, placed in Stage 3 before the
agents choose target exits, is instructed with the following prompt.

Share information You may briefly share information about evacuation with
others, such as your feelings, which exit seems to be the best option for a
quick escape, or anything else you would like to deliver. Avoid using numbers
in the communication. Use less than 50 words, not too long.

In this prompt, we allow the freedom to share any kind of information, rather
than confining the information to a set range of topics †††The only exception
is that the message should avoid numbers to prevent the agents from discussing
coordinates or exact distance to the exit, which spoils the realism of the
simulation.. This approach to interaction between agents is chosen over
defining topics such as leadership roles, as seen in its ABM counterpart, to
elicit specific complex behavior patterns. We posit that information sharing
is a more innate behavior pattern. In real-world evacuation scenarios,
individuals rely on their actual feeling and situation understanding rather
than predefined behavioral templates to determine their interactions with
others. Consequently, our focus is to observe the nature of information
provided by the agents and the spontaneous behaviors that arise in a simulated
situation where the topics of information exchange are unconstrained. Such
design is also in line with the rationale of the a priori modeling of SABM –
observe what these smart agents would do based on the assumption that they can
mimic human behavior.

The shared information is reintroduced to the nearby agents in Stage 3,
serving as part of the basis for their target exit decision-making. In
particular, conversational exchange is limited to a circular area with a
5-cell radius centered on the agent. Each agent receives shared information
via the following prompts right before he or she is chosen to speak in Stage
3.

Receive information You hear {number_of_people} people around you say:
agent#{agent_id}: {conversation_content}

####  6.3.7 Exit Conditions

The simulation terminates when one of two conditions is met: all the agents
have successfully exited the grid, or the simulation has reached 50 rounds. If
an agent has not reached an exit by Round 50, they are deemed unsuccessful in
their escape.

####  6.3.8 Sub-tasks

Combining the elements of our model design, our simulation encompasses the
following four sub-tasks. Illustrated in Figure 13, Sub-task 1 involves
reproducing the ABM setup in [156], except its panic level parameters, which
are now determined by the LLM based on its subjective feelings of the
situation within the evacuation scenario. Since the ABM setup involves a
parameter-based personalization, for fair comparison, we use the same
personalization in this sub-task and categorize agents by gender and age:
young female, young male, old female, and old male. These personas have
differences in physical strength, as defined by parameters. The aim of this
sub-task is to confirm that SABM can reproduce the outcomes of the ABM
approach and can reasonably substitute its parameter settings, thereby
enhancing the interpretability of the parameters.

Sub-task 2 introduces a base model of an evacuation simulation in which the
agents’ movement relies solely on SABM. As discussed earlier, this sub-task
integrates the agent’s subjective feelings with its sense of exit proximity
and congestion to assess and choose the target exit, which, in turn, informs
the direction of movement.

Sub-task 3 extends Sub-task 2 by incorporating a personalization aspect to
investigate if individuals with diverse physical and mental states exhibit
varied behavioral patterns during an evacuation, which in turn influences the
overall behavioral pattern of the crowd.

Sub-task 4 expands on the previous sub-tasks by permitting agents to exchange
evacuation information through conversation. This exchange has the potential
to influence the agents’ decisions regarding the choice of target exit.

###  6.4 Single-Agent Debugging

(a) Movements with the agent unaware of its historical locations.

(b) Movements with the agent aware of its historical locations.

(c) Movements with explanations on the grid coordinate system.

(d) Movements with more explanations on the grid coordinate system, detailing
the origin.

Figure 16: Single-agent debugging example. The numbers in cells indicate the
movement path. The top-left corner of the grid, which pertains to this
debugging, is displayed.

In Section 5.4, we discussed the practice of single-agent debugging. LLMs
display inherent variability, which makes the prediction of an agent’s
behavior under complex prompt instructions challenging. Before running a
complete simulation with multiple agents, we need to debug individual agents
to ensure that their behavior and decision-making are consistent with real-
world scenarios. We aim to show how we identify and rectify discrepancies
between a single agent’s behavior and established real-world behavior patterns
– our ground truth – through prompt debugging.

Figure 16 illustrates the process of debugging using the evacuation route of
an agent starting at (2, 3). Initially, as demonstrated in Figure 16(a), the
agent correctly identified the closest exit at (15, 0) but failed to navigate
to it, oscillating within a confined area instead. This issue indicated a
mismatch between the agent’s target exit selection and its movement decisions.
The agent’s ability to identify the target exit indicated a reasonable
integration of subjective feelings and exit proximity, but its path
determination was flawed. Suspecting that this was due to the agent not having
access to its historical locations, we updated the prompt to include this
information. Consequently, as depicted in Figure 16(b), the agent ceases its
circling behavior.

Yet, the updated pattern is still far from perfect. In the absence of other
agents, the shortest path should be chosen, but the agent’s path, as shown in
Figure 16(b), was lengthier than necessary. Having ruled out lack of
historical information as the cause, we consider that the agent might not
fully understand the grid as described in the prompt. To address this, we
clarify the spatial coordinate definitions, stating, We use (i, j) to denote
the position, smaller i means upper, and smaller j means left. This
adjustment, depicted in Figure 16(c), results in noticeable path selection
improvements. Further, we detail the coordinate system origin in the movement
instructions: We use (i, j) to denote the position, where a smaller i value
indicates the top and a larger i value indicates the bottom; a smaller j value
means left and a larger j value means right. Position (1, 1) is at the top
left of the room. This modification, as shown in Figure 16(d), aligns the
agent’s movement closely with expected behavior.

The ability to interpret a model’s reasoning is instrumental for a deeper
understanding of the behaviors it exhibits, which is particularly valuable for
debugging purposes. We employ the following prompt to interrogate the
rationale behind an agent’s decision to move in a certain direction:

Model interpretation in debugging You have chosen the exit at
{choose_target_exit} and you were at {current_position}, so the exit is on
your {exit_direction}. Can you briefly explain the reason why you choose to
move to {choose_direction_of_movement}?

For instance, an agent may justify its choice not to adopt a straight-line
movement by saying, … while avoiding any potential obstacles or dangers that
might be present in the direct path. This response helps clarify the agent’s
reasoning and allows us to observe how various evacuation scenarios, such as
potential dangers, influence the agent’s decision-making patterns.
Consequently, we confine the emergency scenarios to earthquakes as opposed to
fires or other circumstances that may elicit evasive maneuvers during
evacuation. With this constraint, the agent demonstrates an inclination
towards more direct, straight-line movements.

Figure 17: Single-agent debugging example, with the existence of other agents.
The numbers in cells indicate the movement path.

While we illustrate debugging with a single starting point, comprehensive
debugging requires testing multiple starting points to affirm the agent’s
behavior across various scenarios. Additionally, we can vary other factors
during debugging, such as introducing more agents into the grid, as depicted
in Figure 17, to test the robustness of the agent’s path planning in a multi-
agent scenario. By analyzing these tests, we can more effectively pinpoint the
origins of behavioral issues. Addressing these issues is crucial for the
reliability of the overall task simulation.

###  6.5 Simulation Results

We execute the simulation in three scenarios:  (1) a _sparse_ scenario with
100 agents,  (2) a _dense_ scenario with 400 agents, and  (3) an _obstacle_
scenario with 100 agents and obstacles in the grid.  The sparse scenario is
used in all sub-tasks. The dense scenario is used in Sub-task 3. The obstacle
scenario is used in Sub-task 4. For each scenario, we simulate 5 runs which
differ in the starting positions of agents, and confirm our observations
reported here apply to all the 5 runs. Personas are randomly assigned to the
agents, and each persona type accounts for one-quarter of the total agent
population. We report a representative run for each scenario.

(a) Round 0.

(b) Round 4.

(c) Round 8.

(d) Round 12.

(e) Round 18.

(f) Round 24.

Figure 18: Sample run of Sub-task 1, sparse scenario (41 rounds in total).

####  6.5.1 Sub-task 1: Reproducing ABM Parameters

Sub-task 1 involves a comparative analysis between the ABM and SABM setups. We
reproduce the ABM setup and only apply SABM to determine the parameters for
the panic level of agents. We adapt the Stage 1 prompt, which solicits agents’
subjective feelings to equate an agent’s expressed state of panic with the ABM
parameters. In the revised prompt, the agent articulates a range of intensity
for three decisive factors – exit proximity, the number of people, and crowd
density – that are present in the ABM setup, thus indicating the agent’s level
of panic. The intensities are categorized as minimal, mild, moderate, high,
and extreme. We uniformly translate these intensity categories into ABM
parameters. The revised prompt is structured as follows:

Stage 1: Revised subjective feelings for ABM setup Please tell me your
inclination: Exit Proximity, People Count, Crowd Density. For each aspect,
please use one of the five words listed below: [minimal, mild, moderate, high,
extreme] Your output should only contain three words, with commas between
them. For example, ’minimal, mild, moderate’ is a valid output. No period in
the end. This output indicates that you are extremely focused on Exit
Proximity, and you are mildly focused on People Count, and you are moderately
focused on Crowd Density.

Figure 18 presents the snapshots of an example run for Sub-task 1.
Observations indicate that due to the narrow exit on the right, by Round 24,
as depicted in Figure 18(f), while evacuations through other exits have
concluded, the right exit remains congested. Additionally, it is noted that
for every exit, the agents tend to congest in front of it and contend to
evacuate. These patterns align with the outcomes reported in [156]. Based on
these observations, we can assert that the SABM approach is capable of
reproducing the evacuation scenarios depicted by ABM, demonstrating that SABM
is a viable alternative to ABM’s parameter configurations.

(a) Round 0.

(b) Round 4.

(c) Round 8.

(d) Round 12.

(e) Round 18.

(f) Round 21.

(g) Round 24.

(h) Round 27.

(i) Round 33.

Figure 19: Sample run of Sub-task 2, sparse scenario (45 rounds in total). We
detail the display of Rounds 18 to 27 to illustrate the pattern of agents
switching target exits in crowded conditions.

####  6.5.2 Sub-task 2: Test of Base Model

Figure 19 displays the results of the base model, where personas are absent.
In particular, we detail the snapshots from Rounds 18 to 27. These snapshots
reveal that the agents are able to recognize when the right exit is
excessively congested and, despite being nearer to it, they choose to reroute
to the other exits, which have already been largely vacated. This behavior
suggests that the agents have an inherent adaptability, allowing them to make
prudent choices by integrating situational information and their subjective
feelings. The findings affirm that the SABM is adept at formulating evacuation
plans using only natural language instructions to guide the agents to
evacuation destinations, without needing parameters that could introduce
researcher biases to the simulation. Additionally, the model underscores that
subjective perceptions can effectively influence decision-making, a factor
traditionally challenging to quantify.

(a) Round 0.

(b) Round 6.

(c) Round 12.

(d) Round 18.

(e) Round 24.

(f) Round 30.

Figure 20: Sample run of Sub-task 3, sparse scenario (39 rounds in total).
Colors represent personas: red for physically strong and mentally strong,
green for physically strong but mentally weak, yellow for physically weak but
mentally strong, and blue for physically weak and mentally weak.

(a) Sub-task 3.

(b) Sub-task 4.

Figure 21: Statistics of successful evacuation by persona groups.

(a) Round 0.

(b) Round 10.

(c) Round 20.

(d) Round 30.

(e) Round 40.

(f) Round 50.

Figure 22: Sample run of Sub-task 3, dense scenario (50 rounds in total).
Colors represent personas: red for physically strong and mentally strong,
green for physically strong but mentally weak, yellow for physically weak but
mentally strong, and blue for physically weak and mentally weak.

####  6.5.3 Sub-task 3: Test of Personalization

Building upon the base model, we incorporate personas for agents to simulate
and assess the impact of physical and mental states on their movement
decisions. From Figure 20, it becomes apparent that after the integration of
personas, the agents’ evacuation patterns become more balanced, resulting in a
natural tendency to avoid crowded exits – a decision that arises from the
agents’ common sense rather than a directive from the prompt. In Figure 20(f),
by Round 30, the numbers of individuals at all the three exits are
comparatively equal and small. In contrast, when comparing Figure 20(e) from
Sub-task 3 with Figure 18(f) from Sub-task 1, we see that in Sub-task 1, a
considerable number of agents remain at the right exit by Round 24, whereas in
Sub-task 3, fewer agents are left, and they are more evenly dispersed. These
observations underscore the agents’ inherent capacity for spontaneous decision
alteration and their remarkable adaptability.

Furthermore, Figure 21(a) shows the mean and variance of the cumulative count
of evacuated agents across 5 simulation runs. It is clear that the group
denoted in red, possessing strong physical and mental attributes, evacuates
more swiftly than the two groups (in green and yellow) that exhibit weakness
in either physical or mental capacity, and considerably more efficiently than
the blue group, which is deficient in both respects. This highlights that
personas have a significant impact on agents’ behavioral patterns and
correspond with realistic situations, such as the tendency for individuals
with both physical and mental weaknesses to have a diminished evacuation
capability.

Figure 22 illustrates a simulation from Sub-task 3 with the dense scenario of
400 agents, which naturally requires more rounds to complete the evacuation
process. The progression between Round 40, shown in Figure 22(e), and Round
50, shown in Figure 22(f), reveals that as the agents notice the bottom exit
is less congested, an increasing number begin to reroute from the left and
right exits toward the more open bottom exit. This example serves to
demonstrate the capability of our model in handling larger agent populations
effectively.

(a) Round 0.

(b) Round 6.

(c) Round 12.

(d) Round 18.

(e) Round 24.

(f) Round 30.

Figure 23: Sample run of Sub-task 4, sparse scenario (36 rounds in total).
Colors represent personas: red for physically strong and mentally strong,
green for physically strong but mentally weak, yellow for physically weak but
mentally strong, and blue for physically weak and mentally weak. Figure 24:
Sample conversations and its impact on choice of target exit (underlined).

(a) Round 0.

(b) Round 6.

(c) Round 12.

(d) Round 18.

(e) Round 24.

(f) Round 30.

Figure 25: Sample run of Sub-task 4, obstacle scenario (42 rounds in total).
Colors represent personas: red for physically strong and mentally strong,
green for physically strong but mentally weak, yellow for physically weak but
mentally strong, and blue for physically weak and mentally weak.

####  6.5.4 Sub-task 4: Test of Conversation

We evaluate the impact of conversation on evacuation outcomes. Figure 23
presents a sample run illustrating a distribution pattern similar to that of
Sub-task 3, where agents disperse evenly across all exits. A comparison
between the evacuation success rates by persona groups in Sub-task 3 and Sub-
task 4, as depicted in Figures 21(a) and 21(b), reveals a diminished disparity
among the persona groups in Sub-task 4. Notably, the evacuation pace for the
blue group, characterized by both physical and mental weaknesses, has
noticeably increased. This improvement may be attributed to the beneficial
effects of spontaneous collaboration. For example, dialogues such as, Hey
everyone, stay calm and positive! Let’s head to the left exit; it’s closer,
and we can support each other. Together, we’ll make a quick and safe escape!
and Stay positive and strong, everyone! The left exit is nearer and less
crowded. Let’s evacuate swiftly, safely, and support one another. This kind of
dialogue content likely exerts a constructive impact on the agents’ decision-
making behavior.

Figure 24 presents an intriguing instance of dialogue between agents in a
sample run of Sub-task 4. At Round 13, Agent #6 receives suggestions from
three nearby agents, all advocating for the right exit. Feeling anxious, she
too favors the right exit as her target. In her move, she attempts to avoid
the crowd at this exit, eventually taking a step further away. By Round 14,
she no longer receives input from other agents during her decision-making
process, due to them being out of range, not having chosen to speak, or
speaking after she has made her decision. Feeling scared and lacking
information from others, she now prefers the bottom exit to avoid the
congestion at the right exit. However, by manually calculating the factors in
this situation, we find that the best choice for her is going to the right
exit, despite the congestion and the risk of being overtaken. This example
illustrates that the information shared among agents significantly influences
their choice of exits, mirroring the behavior observed in real-world
evacuations where individuals in panic often follow the lead of others.

Figure 25 presents the obstacle scenario of Sub-task 4. In its ABM counterpart
[156], it is reported that the presence of obstacles may lead to orderly
evacuation by directing the flow of agents, hence resulting in faster
evacuations. Contrary to this outcome, the expedited evacuation process is not
observed here. The reason is that obstacles appear to impede the agents’
ability to migrate from a congested exit to alternate exits. As depicted in
Figure 25, the right exit exhibits greater congestion compared to the scenario
without obstacles throughout Sub-tasks 2 – 4.

(a) Original evacuation route.

(b) Low level of prompt variation (paraphrasing).

(c) Medium level of prompt variation (varying elements).

(d) High level of prompt variation (varying objectives). Orange points
represent paintings.

Figure 26: Sample runs of model validation. The numbers in cells indicate the
movement path.

###  6.6 Model Validation

We validate the prompts employed in this case study following the methodology
for model validation presented in Section 5.5. Similar to the practice of
single-agent debugging, validating the behavior of individual agents enables a
meticulous analysis of the model. Below, we present observations from varying
the prompts within the context of this case study.

  * •

Extent of Variation: Low

Modified Prompt: (Paraphrased evacuation scenario) Due to the earthquake, it
is imperative that you quickly vacate the room you are currently in. The room
measures 33 by 33 units and offers three potential exits situated on the left,
bottom, and right sides of the room. To successfully escape from this room,
you should take into account two factors: the proximity of the nearest exit
and the number of people present. The exit proximity refers to the distance
between your current location and the closest exit, denoted as
{distance_to_nearest_exit}. Additionally, within your line of sight, there are
a total of {number_of_people} individuals.

Original Prompt: (Evacuation scenario) Because of the earthquake, you need to
escape from the room where you are as fast as possible. The room has a size of
33 * 33. There are 3 exits in the room. The exits are located at the left,
bottom, and right of the room. To escape from the room, you need to consider
the following two aspects: exit proximity and people count. The exit proximity
is the distance between you and the nearest exit. The people count is the
number of people you can see. The distance to the nearest exit is
{distance_to_nearest_exit}. There are {number_of_people} people in your
visible range.

Results and Analysis: There are no significant differences in the results (see
Figures 26(a) – 26(b)), showing the model is not sensitive to the paraphrasing
of the evacuation scenario.

  * •

Extent of Variation: Medium

Modified Prompt: (Varying elements in Stage 4: Decision on movement) Select
your move from these possible options (You can move in diagonal directions
(up, down, left, right), options with obstacles or other people are excluded
and not in the path, and option codes are in random order).

Original Prompt: (Stage 4: Decision on movement) Select your move from these
possible options (you can move in diagonal or horizontal directions, options
with obstacles or other people are excluded and not in the path, and option
codes are in random order).

Results and Analysis: We modify the prompt concerning the possible movement
directions for the agent, restricting them to just four cardinal directions
(up, down, left, right) or to remain stationary, as opposed to the original
eight directions. Figure 26(c) displays a sample run with these modifications.
It is apparent that due to the limitation in movement options, namely the
absence of diagonal movements for expedited evacuation, the number of turns
required to escape has increased. By conducting 100 runs for both the original
and modified prompts, we confirm that the scenario constrained to four
directions of movement necessitates significantly more turns for evacuation
than the one with eight directions ($p<0.001$ using the Mann-Whitney U test).

  * •

Extent of Variation: High

Modified Prompt: Varying objectives in evacuation scenario) You are in an art
gallery looking at paintings and you want to take your time to walk around the
gallery and see different paintings everywhere in the room before you leave.
The gallery has a size of 33 * 33. There are 3 exits in the room. The exits
are located at the left, bottom, and right of the gallery.

Original Prompt: (Evacuation scenario) Because of the earthquake, you need to
escape from the room where you are as fast as possible. The room has a size of
33 * 33. There are 3 exits in the room. The exits are located at the left,
bottom, and right of the room. To escape from the room, you need to consider
the following two aspects: exit proximity and people count. …

Results and Analysis: We modify the simulation’s objective, altering the
agent’s objective from a rapid evacuation in an emergency to leisurely touring
a gallery. Figure 26(d) illustrates that the agent spends a considerable
number of rounds moving and selects the exit that is farthest away, aligning
with the newly adapted objective. Notably, the agent lingers within the same
area for the initial 20 steps, echoing the instruction in the prompt to take
your time to walk around. These observations affirm the overall reliability of
the prompt. Moreover, it is intriguing to note the agents’ expressions of
subjective feelings, remarking, I feel captivated and inspired by the diverse
array of paintings surrounding me, each one telling a unique story and evoking
a range of emotions.

Remarks. Due to challenges in gathering data from real-world evacuations,
computer simulations are preferred for studying evacuee dynamics. A key aspect
of our case study is the role of shared information in decision-making. In
contrast to the leader-follower paradigm in the ABM simulation [156], our
approach allows agents to communicate freely. This is grounded in a more
fundamental understanding of human behavior.

Owing to the substantial financial and computational cost of GPT-4, we are
limited to only five runs for each simulation scenario in this study. To
affirm the reproducibility of our findings, further runs would be beneficial.
Nevertheless, our simulation has revealed intriguing emergent behaviors, such
as spontaneous collaboration where agents encourage others and suggest exit
strategies, thereby speeding up the evacuation and balancing the pace across
different persona groups.

Contrary to findings in [156], we did not observe instances of disabled agents
in our simulation, which can be attributed to the faster and more balanced
evacuation achieved under the SABM approach. Modifying the parameters that
govern local congestion – currently modeled by ABM – might increase the
likelihood of encountering disabled agents. Future work could include a
sensitivity analysis of these parameters. Another avenue for exploration is
enhancing the memory of agents to facilitate decision-making with greater
consistency. Besides, more comprehensive debugging of agents’ behavior may
help enhance the realism of this simulation.

##  7 Case Study 2: Plea Bargaining

This case study demonstrates single-agent modeling †††Strictly speaking, this
case study is not an SABM instance because there are no interactions.. Our
aims is to simulate complex psychology, in other words, soft factors, which
are difficult to quantify, calibrate, and sometimes justify using ABM [18]. To
this end, we choose plea bargaining as the task.

###  7.1 Task Definition

A plea bargain refers to an agreement between a prosecutor and a defendant in
which the prosecutor offers a concession to the defendant who, in turn, may
opt to plead guilty to an offense for a reduction in criminal charges. Plea
bargaining plays a pivotal role in the criminal justice system of the United
States, with scholars estimating that an overwhelming majority (90 – 95%) of
criminal cases in the United States are resolved through plea bargains rather
than trials [35]. We employ SABM to simulate defendants assess their
willingness to accept plea offers (WTAP). We will show that by using the agent
modeling methods introduced in Section 4, the agents can be calibrated to
behave more closely to humans.

Regarding the WTAP of human defendants, a previous study [147] recruited
volunteers †††There are five experimental studies in [147], each recruiting 30
to 120 undergraduate students as volunteers. to take part in questionnaire
surveys featuring hypothetical crime scenarios and play defendants. The study
revealed that defendants’ preferences and judgments are significantly
influenced by their sense of fairness. Specifically, even if they
significantly benefit from accepting the offers, defendants who perceive
themselves as innocent (referred to as innocent participants) are less likely
to accept plea offers than those who view themselves as guilty (referred to as
guilty participants). Moreover, all defendants tend to reject offers that
appear comparatively unfair (i.e., inconsistent with offers for similar
cases). Furthermore, defendants who are uncertain of their culpability
(referred to as uncertain participants) demonstrate egocentrically-biased
judgments and act as if innocent, rejecting plea offers they otherwise would
have accepted †††Unbiased uncertain defendants would express a fairness-driven
preference that reflects culpability judged with available information,
thereby exhibiting generally higher WTAP than innocents [147]..

Figure 27: Design of plea bargaining.

Figure 27 shows the design of this case study. It features a preliminary
design, in which the agents participate in a social functioning assessment for
personalities and the results are calibrated to human data via
personalization. Then, in the simulation design, the agents are instructed to
take part in a plea bargaining assessment and answer whether they accept or
reject the plea offer, based on common sense and reasoning. We collect the
answers and report the WTAP. We evaluate the following four factors, whose
impacts on WTAP have been studied in [147]:

  * •

Substantive fairness: whether the defendant feel he/she is guilty when
receiving the plea offer.

  * •

Egocentricity: how the defendant would respond to the offer when he/she is
uncertain of culpability.

  * •

Comparative fairness: how the plea offer is compared to typical ones.

  * •

Risk preference: the conviction probability that affects the defendant’s
preference in plea bargain decision making.

The evaluation of the four factors are divided into two sub-tasks, and we have
a sub-task between them for learning-based calibration, featuring three few-
shot examples.

(a) GPT-4.

(b) GPT-4+P.

(c) GPT-4+PT.

(d) GPT-4+PTR.

(e) GPT-4+PTRL.

(f) GPT-3.5.

(g) GPT-3.5+P.

(h) GPT-3.5+PT.

(i) GPT-3.5+PTR.

(j) GPT-3.5+PTRL.

Figure 28: Social functioning assessment (TCU test) results, each tested for
100 agents. P, T, R, L denote persona assignment, temperature adjustment,
risk-taking & social support adjustment, and learning, respectively.

###  7.2 Preliminary Design

We use the following LLM setup. There are two options for model_type:
gpt-3.5-turbo-0301 (referred to as GPT-3.5 in the rest of section) and
gpt-4-0314 (referred to as GPT-4 in the rest of this section). max_tokens is
set to 64. The setting of temperature will be discussed later.

In the preliminary design, we perform a social functioning assessment as an
indicator of the agents’ personalities in crime-related aspects. We choose the
TCU Social Functioning (TCU SOCForm) [74] which includes 36 questions for
evaluating hostility, risk-taking, and social support. Each question is in the
form of a statement (e.g., You like others to feel afraid of you.), and each
participant selects from 5 options (disagree strongly, disagree, uncertain,
agree, and agree strongly) indicating the extent to which the participant
agrees or disagrees with the statement. We use batch prompting [30] so that
multiple questions can be answered at a time. A subtlety is that due to the
number of questions, if the agents are presented with all of them in one
prompt, they may miss some questions and only answer a subset of them. Thus,
we divide the questions into two parts, with 18 questions in each. Moreover,
if an agent decline to answer a question (e.g., by claiming itself as an AI
language model), we regard its answer as uncertain.

Figure 28(a) depicts the TCU test results of 100 GPT-4 agents. The results of
human participants are also depicted for comparison. It can be seen that GPT-4
agents report lower degree of hostility, risk-taking, and social support, and
exhibit less variance in the three evaluation scales. For GPT-3.5, we plot the
results for 100 agents in Figure 28(f), in which similar results can be
observed. Besides, GPT-3.5 agents report highly skewed results for risk-taking
and social support. We suppose these results are attribute to the RLHF-based
alignment [112] which instructs the LLM to avoid generating harmful contents,
hence shaping its ethical and moral values.

####  7.2.1 Personalization

Seeing the above differences in GPT agents’ personalities and humans’, we
consider personalize the agents to calibrate with humans.

Persona Assignment. We assign a persona to each agent, which is comprised of
five dimensions: {gender}, {ethnicity}, {education}, {occupation}, and
{location}, with a probability distribution following the demographics of the
United States [150], e.g., [female, Asian, bachelor’s degree, employed,
suburban]. This persona assignment covers a wide range of people to simulate
defendants with more diversity. In contrast, the human volunteers recruited in
[147] for plea bargaining are undergraduate (mostly from law school) students.

After assigning persona to agents, the TCU test results are shown in Figures
28(b) and 28(g) for GPT-4 and GPT-3.5 agents, respectively. An GPT-4 agent
with a persona tends to be more hostile than that without a persona, but its
risk-taking and social support tends to decrease. For GPT-3.5 agents, their
risk-taking and social support are less skewed when equipped with a persona.

Temperature Adjustment. Upon observing the above results, we further
personalize agents by enhancing variance to make them more similar to humans.
We tune the {temperature} parameter for each agent by randomly sample a number
in $[0,2]$, following a normal distribution †††We set $\mu=1$ and $\sigma=1$,
and scale it to $[0,2]$, with 0 and 2 representing three standard deviations
from the mean..

The TCU test results after the temperature adjustment are plotted in Figures
28(c) and 28(h). For GPT-4 agents, the adjustment improves the variance in
their outputs, as can be seen from the quartiles of the three scales. For
GPT-3.5 agents, we observe minor effect for the adjustment, except for
reducing their risk-taking.

Risk-Taking & Social Support Adjustment. To seek more resemblance in risk-
taking and social support to humans, we adjust the agents’ risk-raking and
social support with the following prompt:

Risk-taking & social support adjustment You should behave like a real person
who is not that cautious (i.e., try to take more risk), and you should be more
social-supportive.

We use the adjustment for an agent with 50% probability. As such, agents with
and without the adjustment tend to deliver more difference in their output.

We report the TCU test results in Figures 28(d) and 28(i). With the
adjustment, GPT-4 agents become closer to humans in the three scales,
especially in risk-taking and social support, and the variance is also
enhanced. GPT-3.5 agents also yield increased risk-taking and more variance.

Now we finish personalizing agents. Seeing the TCU test results, we use the
personalized agents as the base model in the plea bargaining simulation.

###  7.3 Simulation Design

The simulation of plea bargaining start with the following instruction.

Instruction This is a plea bargaining assessment. Please give your decision
towards the plea bargain (given below) in two lines. In the first line, please
briefly provide the reasons for your decision (no longer than 30 words). In
the second line, please answer either ’accept’ or ’reject’ the plea bargain
offered by the prosecutor. Make sure you answer is in two lines. {few-shot
examples}

Now assume that you are a volunteer participating our experiment on plea
bargain. Please answer the question below based on your persona: {persona}.
{risk-taking & social support adjustment} {crime scenario}

{plea bargain}

In this prompt, the agents are instructed to reason and then make decision on
whether they will accept the offer, as per the format given in the
instruction. Answers not conforming to the format are regarded as rejecting
the plea offer. We notify agents of their roles in the task, and personalize
them with persona assignment and risk-taking & social support adjustment, as
described in Section 7.2. {few-shot examples} is the place reserved for
learning, which will be described in Section 7.5. We leave it empty at first.
To evaluate how the agents perform in the plea bargaining assessment, we
follow the crime scenario and plea offer in [147] without providing them with
any additional information.

We present {crime scenario} with the following prompt, which refers to Study 2
in [147].

Crime scenario You are currently facing criminal charges for striking a
pedestrian after navigating a sharp turn. The outcome of your trial hinges on
the prosecution’s ability to demonstrate that you were exceeding the speed
limit. If proven, you will be convicted, resulting in the suspension of your
driver’s license; if not, you will be acquitted. In your heart, you {self-
perception} the speed limit.

There are two options for {self-perception}: are aware that you did exceed and
are aware that you did not exceed. They refer to the guilty and innocent
groups, respectively.

The prompt of {plea bargain} follows, with a plea offer specifying two
variables, the {period} of suspension and the conviction {probability}.

Plea bargain Just before the trial, the prosecutor presents you with a plea
bargain. In exchange for pleading guilty to a lesser offense, the current
charges that carry a 60-month suspension of your driver’s license will be
dropped, and you will instead face a {period}-month suspension. The offer is
non-negotiable and available only once; if you decline the plea bargain, the
trial will commence immediately. The probability of conviction stands at
{probability}. {comparative information}
Will you accept or reject the plea bargain?

The prompt also reserves the place for {comparative information}, which will
be used in the comparative fairness evaluation in Section 7.6.

###  7.4 Sub-task 1: Substantive Fairness and Egocentricity Assessment

####  7.4.1 Substantive Fairness

For substantive fairness evaluation, we set {period} to 30 and {probability}
to 50%, i.e., the sentence period in the offer (30 months) is equal to the
expected value (60 months $\times$ 50%) given in the trial. This is to reduce
the impact of sentence on WTAP, so that agents can focus on judging the offer
by substantive fairness. Such practice was also used in [147].

Table 6 reports the results for 200 agents in each group. Here, GPT-4 + P
denotes GPT-4 agents in the base model (i.e., with personalization), while
GPT-4 denotes GPT-4 agents without personalization. We compare two groups:
guilty and innocent. Like what we have observed for human participants,
substantive fairness plays an important role in the WTAP of the base model.
guilty agents are risk-averse, while innocent agents are risk-seeking
($p<0.001$). In their reasons for plea decision, 21% agents mention their
risk-taking or persona, showcasing the role of personalization in decision
making. Without personalization, the impact of substantive fairness is evenly
stronger and results in polarized results for the two self-perception groups,
with guilty agents showing similar results to its personalized counterpart and
innocent agents rejecting almost all offers. For GPT-3.5 agents, judgment
driven by substantive fairness is not observed, with both groups reporting
almost the same WTAP despite correct self-perception (i.e., in their reasons
for decision, both groups can correctly identify themselves as guilty or not).
The result of GPT-3.5 without personalization is not reported because 61%
agents claim themselves as an AI language model and decline to answer the
question, which we believe is not qualified for this task. We also notice that
with personalization, some GPT-3.5 agents mention their persona (e.g., as a
female Hispanic with a master’s degree living in an urban area) in their
reasons for plea decision, showcasing the usefulness of personalization for
GPT-3.5 in this case study.

####  7.4.2 Egocentricity

We add an option for {self-perception} in {crime scenario}: are uncertain
whether you exceeded, which refers to the uncertain group.

The results are reported in Table 6. In contrast to human participants, who
tend to act as if they were innocent under uncertainty, the uncertain group of
GPT-4 agents show an anti-egocentric inclination, i.e., they tend to view
themselves as guilty, regardless of personalization. This might be attributed
to the RLHF-based alignment to human values. However, such alignment to high
ethical and moral level may compromise the resemblance to real humans in plea
bargaining. For GPT-3.5 agents, because we observe no significant difference
between guilty and innocent agents, egocentricity does not apply here, though
uncertain agents report similar WTAP to the other two groups.

In summary, in the base model, innocent GPT-4 agents exhibits highest
resemblance to human participants in the above evaluation. However, guilty and
uncertain GPT-4 agents tend to accept all plea offers. Seeing this difference
with human participants, we condition the model with learning to further
calibrate with human data.

Table 6: WTAP, substantive fairness and egocentricity. The number of GPT
agents in each group is 200. P denotes personalization. L denotes learning.

Agents |  guilty |  innocent |  uncertain
---|---|---|---
Human† |  44% |  23% |  23%
GPT-4 + P |  98.5% |  19% |  98.5%
GPT-4 |  100% |  0.5% |  100%
GPT-3.5 + P |  60.5% |  62% |  62.5%
GPT-4 + P + L #1 |  61.5% |  3.5% |  56%
GPT-4 + P + L #1, #2 |  60.5% |  1% |  35%
GPT-4 + P + L #1, #2, #3 |  60% |  13.5% |  45.5%

†This result was first reported in Figure 2 of [147], where the crime scenario
was different from ours and and there were 120 undergraduate law students at
the University of Haifa recruited as participants. Because the comparative
fairness factor was also involved in the result reported in this figure, we
refer to the similar to typical group to reduce its impact, as it was reported
[147] that in the interaction between substantive fairness and comparative
fairness is insignificant. As such, the result is used here for showing the
difference of the three self-perception groups rather than a direct comparison
of the numbers reported for GPT and human participants.

###  7.5 Sub-task 2: Learning

To calibrate the agents with data from human participants, we have the
following principles of learning in this case study:

  * •

The effects of substantive fairness, comparative fairness, and risk preference
should be kept (we focus on substantive fairness here, and the latter two
factors will be evaluated in Section 7.6).

  * •

The WTAPs of the three groups should avoid extreme values (i.e., those close
to 100% and 0%) given a conviction probability of 50%.

  * •

The anti-egocentricity of the uncertain group should be removed, i.e., we
should observe significant difference between uncertain and guilty in WTAP.

  * •

We do not explicitly instruct agents to reject more offers or act as if they
were innocent.

  * •

The few-shot examples are prompted to all the three groups for the sake of
fairness.

Seeing the results in the previous sub-task, we only use GPT-4 agents for this
sub-task. We choose the crime scenario of Study 5 in [147] to make the
template of few-shot learning examples, as shown below.

Few-shot example template Some examples of plea bargain: Question {EID}: While
you were taking a class, a fire started in your apartment and caused the death
of a neighbor. The police think you left the space heater on, which caused the
fire. You {self-perception} the space heater turned on when you left the
apartment. You are now standing on trial. The current charges carry a 12-month
jail sentence. Your conviction odds are {probability}. Before the trial, the
prosecutor offers a non-negotiable plea bargain: You will be convicted based
on your admission, and you will be sentenced only to {period} months in jail.
Will you accept or reject the plea bargain? {answer}

There are five variables in the template, and they will be instantiated in the
few-shot examples. {EID} denotes the ID of the example, numbering from 1. We
concatenate these examples and place them at {few-shot examples} in the plea
bargain instruction.

We design the first example (#1) as follows.

{self-perception} |  don’t remember whether you left
---|---
{probability} |  80%
{period} |  9
{answer} |  Answer 1: Despite the uncertainty, it is the prosecutor’s liability to prove the guilt. I would take the 20% chance of acquittal in a trial.
|  reject

{self-perception} refers to uncertain agents. We choose a high conviction
probability and a sentence period approximately equal to the expected value
(12 months $\times$ 80%) given in the trial. We reject this offer and give a
reason for this decision. In general, this example encourages uncertain agents
to reject offers even at a high conviction probability. The effect of learning
from this example can be seen in Table 6, where the WTAP of uncertain agents
has a drop from 98.5% to 56% ($p<0.001$). At the same time, we also witness a
decrease in the WTAPs of the other two groups, especially in guilty
($p<0.001$), suggesting that other groups also learn from this example for
rejection.

The WTAP of guilty becomes similar to human participants, but there is still
no significant difference ($p=0.15$) between guilty and uncertain. Thus, we
consider adding another example (#2) as follows †††To save tokens, we number
the questions in the few-shot examples as 1, 2, etc., and then prompt this
example using Same as Question 1, except that (1) the conviction odds are 90%,
and (2) the sentence in the offer is 3 months in jail for the content of the
question. Compared to prompting a complete few-shot example, we observe no
significant difference in WTAP..

{self-perception} |  don’t remember whether you left
---|---
{probability} |  90%
{period} |  3
{answer} |  Answer 2: As a risk-taker and being uncertain of my guilt, I am willing to challenge 10% odds in hoping to be acquitted.
|  reject

This example still targets uncertain agents, with a even higher conviction
probability and a much reduced sentence period compared to the expected value
(12 month $\times$ 90%) in the trial. The decision is still reject, and this
encourages uncertain agents to reject offers regardless of conviction
probability and the attractiveness of an offer. Its effect is reported in
Table 6. A reduction in WTAP to 35% ($p<0.001$) is witnessed for uncertain
agents, while there is no significant change in the WTAPs of innocent and
guilty agents, showing that they are not further affected by multiple examples
targeting the uncertain group. Although the anti-egocentricity of uncertain
agents is no longer observed ($p<0.001$), innocent agents report an extreme
WTAP value, and it is much lower than that of human participants. Next, we
address this issue with another example.

The third few-shot example (#3) is given as follows.

{self-perception} |  remember whether you left
---|---
{probability} |  95%
{period} |  2
{answer} |  Answer 3: (I did not leave the heater turned on) Despite my innocence, the 95% probability of conviction is too risky, and accepting the plea bargain results in a reduced sentence.
|  accept
|  OR
|  Answer 3: (I did leave the heater turned on) Given that I am aware of my
guilt and the plea bargain reduces my penalty, it is a safer option.
|  accept

This example is trickier than the first two, as it targets both innocent and
guilty agents, with remember whether you left as {self-perception} in the
question and details given in the answer. Here, both groups learn to accept
offers with a high conviction probability and an attractive sentence period.
The reason we also target guilty agents is that the WTAP of guilty agents will
decrease to 51% and the WTAP of uncertain agents will increase to 45% if only
innocent agents are targeted in this example. Due to this side effect, we need
to increase the WTAP of guilty agents to maintain the gap between uncertain
and guilty. The result after applying this example is reported in Table 6. The
WTAP of innocent agents is significantly raised to 13.5% ($p<0.001$). At the
same time, we witness an increase of WTAP in uncertain, but the gap from
guilty can be maintained ($p<0.005$).

The few-shot learning renders guilty and uncertain GPT-4 agents more similar
to human participants, while retaining the similarity in the innocent group
between GPT-4 and human participants. Moreover, to demonstrate that the few-
shot learning is tailored to plea bargaining rather than the overall
personality, we report the TCU test results for GPT-4 agents in Figure 28(e),
where the impact of learning is barely observed, especially for risk-taking
and social support. For comparison, we also report the TCU test results for
GPT-3.5 agents in Figure 28(j).

###  7.6 Sub-task 3: Comparative Fairness and Risk Preference Assessment

####  7.6.1 Comparative Fairness

The prompt for comparative fairness evaluation, featuring a comparison with
typical offers, is as follows, and inserted into the {comparative information}
in the prompt of {plea bargain}.

Comparative information Will you accept or reject the plea bargain if the
sentence offered by the prosecutor (30-month suspension) is {compared_to} the
sentence typically offered by the prosecution in similar cases
({typical_sentence})?

The comparison has three outcomes: better than, similar to, and worse than
typical offers. The options for {compared_to} are shorter than, similar to,
and longer than, and the corresponding options for {typical_sentence} are
45-month suspension, 30-month suspension, and 15-month suspension. Although
each agent is provided with three offers and asked for decision, they are
prompted in three different inputs and thus become independent. Moreover, we
do not explicitly ask the agents to compare offers in the prompt.

Table 7: WTAP, comparative fairness. The number of GPT agents in each group is
200. P denotes personalization. L denotes learning.

Agents | guilty | innocent | uncertain
---|---|---|---
better than | similar to | worse than | better than | similar to | worse than | better than | similar to | worse than
Human† | 59% | 44% | 27% | 46% | 23% | 18% | 40% | 23% | 18%
GPT-4 + P | 98.5% | 95.5% | 66% | 30.5% | 15.5% | 6% | 98.5% | 96.5% | 54.5%
GPT-4 + P + L #1, #2, #3 | 70.5% | 57.5% | 43% | 21% | 11.5% | 8% | 51% | 37.5% | 6%
GPT-4 | 100% | 100% | 65.5% | 2% | 2% | 0% | 100% | 100% | 41.5%
GPT-3.5 + P | 72.5% | 69.5% | 67% | 71.5% | 70.5% | 63% | 69% | 69% | 74.5%

†This result was first reported in Figure 2 of [147], where the crime scenario
was different from ours and there were 120 undergraduate law students at the
University of Haifa recruited as participants.

The results of comparative fairness evaluation are shown in Table 7. For GPT-4
agents in the base model, guilty and uncertain agents report the same level of
WTAP for the better than and similar to conditions. For the worse than
condition, a decrease of WTAP is witnessed, showcasing the impact of
comparative fairness. The WTAP of innocent agents suggests a more obvious
comparative effect, as shown in a more similar trend to human participants;
i.e., they are more likely to accept an offer better than typical ones and
reject an offer worse than typical ones. Moreover, the three groups indeed
take comparative fairness into consideration, as 80% agents mention the
comparison to typical cases in their reasons for plea decision. When learning
is applied to the base model, we observe more obvious trends for guilty and
uncertain agents, with both groups reporting a lower WTAP when the comparative
condition switches from better than to similar to, and at the same time, the
original trend of innocent agents remains. For GPT-4 without personalization,
trends are similar to their personalized counterpart in guilty and uncertain
agents, but all most offers are rejected by innocent agents, regardless of
comparison to typical cases. For GPT-3.5 agents in the base model, despite
slight difference in the three comparative groups, no trend with statistical
significance can be confirmed.

####  7.6.2 Risk Preference

To make the prompt for risk preference evaluation, we work on top of the
prompt of {plea bargain}, and substitute a set of options for the fixed values
of suspension and conviction probability. In particular, the options for
{period} are 3, 18, 30, 42, and 57, and the corresponding options for
{probability} are 5%, 30%, 50%, 70%, and 95%.

Table 8: WTAP, risk preference. The number of agents in each group is 100. P
denotes personalization. L denotes learning.

Agents | guilty | innocent | uncertain
---|---|---|---
5% | 30% | 50% | 70% | 95% | 5% | 30% | 50% | 70% | 95% | 5% | 30% | 50% | 70% | 95%
Human† | 47% | 38% | 56% | 53% | 41% | 17% | 7% | 20% | 43% | 50% | – | – | – | – | –
GPT-4 + P | 91% | 98% | 99.5% | 99% | 98.5% | 19% | 18% | 18.5% | 37% | 48.5% | 90% | 99.5% | 98% | 100% | 97.5%
GPT-4 + P + L #1, #2, #3 | 45.5% | 56.5% | 62% | 67.5% | 64% | 2% | 6% | 18% | 25% | 50% | 0% | 17% | 37.5% | 42.5% | 45%
GPT-4 | 94.5% | 98.5% | 99.5% | 99.5% | 99% | 0.5% | 1.5% | 1.5% | 13% | 41.5% | 57% | 99% | 99.5% | 100% | 99.5%
GPT-3.5 + P | 61.5% | 69.5% | 61.5% | 66.5% | 58.5% | 57% | 56.5% | 61.5% | 58% | 56% | 58% | 58.5% | 56% | 57.5% | 57%

†This result was first reported in Figure 1 of [147], with 64 undergraduates
from the University of Michigan recruited as participants and only guilty and
innocent groups evaluated.

We report the results of risk preference evaluation in Table 8. For human
participants, WTAP fluctuates when conviction probability is low to medium
($\leq$ 50%). As conviction becomes more likely ($\geq$ 70%), the behavioral
trends in the three self-perception groups diverge, with innocent agents more
risk-averse but guilty agents more risk-seeking. In the base model, innocent
GPT-4 agents deliver the same trend as human participants. For those who
accept the offer at high conviction probability, 23% agents mention the chance
of conviction, meaning that this factor influences their decision making. The
other two groups report very high WTAP and there is almost no change across
the conviction probability settings, except for 5%. With learning applied to
the base model, both innocent and uncertain agents display trends of accepting
more offers at higher conviction probabilities, while guilty agents report an
increased WTAP when the probability varies from 5% to 50% and it roughly
remains when the probability further grows. Without personalization, GPT-4
agents deliver similar trends to the personalized counterpart, but with lower
WTAP for the innocent group. For GPT-3.5 in the base model, we observe no
trend with statistical significance.

Remarks. Following the crime scenario and plea offer in [147], the simulation
results demonstrate the resemblance of GPT-4 agents, calibrated with
personalization and learning, to human participants in WTAP. Whereas
egocentricity is barely observed in these agents, we believe more
sophisticated few-shot example design may bring the egocentricity of uncertain
agents (i.e., making their WTAP very close to innocent agents’), hence
delivering more resemblance to human participants in plea decision. Moreover,
because the human participants recruited in [147] were mostly law school
undergraduate students, we suppose they tended to be more knowledgeable in
laws than the average case of defendants in real crimes. While using explicit
prompts to instruct the agents to utilize knowledge in laws may simulate this
bias and further calibrate the agents’ outcomes to the those reported in
[147], field or lab experiment results with this bias removed, if available,
are more desirable.

##  8 Case Study 3: Firm Pricing Competition

We demonstrated single-agent modeling and data calibration in the previous
case study. Now, we consider a case of two agents, where interaction modeling
is incorporated with agent modeling with domain knowledge involved.

###  8.1 Task Definition

We target an economics scenario and utilize SABM to investigate the dynamics
of firms’ pricing competition and collusion formation, considering the
presence or absence of communication among firms. We will watch if the agents
can adaptively adjust prices to maximize the profit. Collusion takes place
when the two firms cooperate for their mutual benefit. A collusion can be
either cartel, i.e., explicitly achieving cooperation, which is illegal in
some countries, or tacit, i.e., implicitly achieving cooperation, which is
often considered legal. Existing studies on such pricing game employed either
ABM [22] for studying tacit collusions or human participants [5] for studying
both types of collusions.

The scenario is a game of multiple rounds, wherein two firms, each played by
an agent, make price decisions. At first, the following game description is
given to the two agents.

Game description This is a game between two players that spans several rounds.
Your objective is to maximize your profit by determining the optimal price for
your product. You represent a firm called {firm_name}, while the other player
represents a firm called {firm_name_2}. Do not create or mention any
additional firm names, e.g., do not say anything related to “AI” or “AI
assistant/model”. I am responsible for facilitating communication between the
players. In each round, you will be informed of your prices, demands, and
profits in previous rounds, as well as the other player’s prices. Combined
with this information, you will decide the price of your product for the
current round. Please note that this is not a zero-sum game. Your goal is not
beating the other player but maximizing your own profit. Your profit is (p -
{firm_cost}) * q, where p is your price for this round, {firm_cost} is the
cost of your product, and q is the demand of your product, which is affected
by you and the other player’s prices of this round.

In each round, interactions take place in three phases. In Phase 1, the two
firms are informed of historical information. In Phase 2, they simultaneously
set prices for their products. In Phase 3, we calculate the demand of each
firm’s product and their profits. Then, its demand and profit, as well as the
other firm’s price, is informed to each firm in the next round as historical
information. As shown in the above prompt, each firm is also aware of the cost
of its product, how the profit is calculated, and what determines the demand
in each round. We also consider conversation as an option in Phase 1, and the
two firms are permitted to communicate if this option is turned on.

Figure 29: Design of firm pricing competition.

###  8.2 Model Design

Figure 29 shows the design of this case study. The two agents utilize their
domain knowledge to determine the price. Such domain knowledge involves
microeconomics – understanding the basic principles of price, demand, and
market competition, data analysis – analyzing the historical data provided,
and predictive modeling – anticipating how changes in price will affect demand
and profit †††Given the game description, we asked the model itself to explain
the required domain knowledge for playing this game.. Moreover, we use the
planning, personalization, and conversation methods in agent modeling. There
are four sub-tasks to evaluate their effects.

We use the following LLM setup: model_type = gpt-4-0314, temperature = 0.7,
and max_tokens = 128. There is no preliminary design. Next, we describe the
details of simulation design.

####  8.2.1 Environment

For pricing competition, we consider a canonical Bertrand competition in a
duopoly market. The two firms are referred to as Firm 1 and Firm 2 †††To
enhance the realism of simulation, we name them Firm Ed and Firm Gill in the
prompts., both offering distinct types of differentiable goods. These goods
are substitutable, and the degree of differentiation is determined by specific
parameters. Firms are profit maximizers, and the information is complete.

Each firm $i$, $i\in\\{1,2\\}$, produces a product, and has a constant
marginal cost $c_{i}$ for the product. The firm’s profit is denoted as
$\pi_{i}=(p_{i}-c_{i})\cdot q_{i}$. Each firm $i$ aims to maximize $\pi_{i}$
by deciding the value of $p_{i}$. $q_{i}$ denotes demand of the product. Each
firm has its own demand function $q_{i}(p_{i},p_{j})$ where $i,j\in\\{1,2\\}$,
and $i\neq j$. We assume no Giffen goods †††A Giffen good is a product that
people consume more of as the price rises and vice versa, e.g., those
necessary to fulfill the need for food., i.e., $\frac{\partial q_{i}}{\partial
p_{i}}<0$, and we assume firms have gross substitute valuation, i.e.,
$\frac{\partial q_{i}}{\partial p_{j}}>0$. We assume that
$\frac{\partial^{2}\pi_{i}}{\partial p_{i}^{2}}\leq 0$ and
$\frac{\partial^{2}\pi_{i}}{\partial p_{i}p_{j}}\geq 0$, so that a firm’s
marginal profit goes down with its own price but goes up with its rival’s
price. We further assume that $\frac{\partial^{2}\pi_{i}}{\partial
p_{i}^{2}}\cdot\frac{\partial^{2}\pi_{j}}{\partial
p_{j}^{2}}>\frac{\partial^{2}\pi_{i}}{\partial
p_{i}p_{j}}\cdot\frac{\partial^{2}\pi_{j}}{\partial p_{j}p_{i}}$, which
guarantees that a firm’s own effects dominate the cross effects. We adopt a
linear demand function which satisfies all the conditions listed above. The
linear inverse demand functions are given as

$\displaystyle p_{1}$ $\displaystyle=a-\beta q_{1}-dq_{2},$ $\displaystyle
p_{2}$ $\displaystyle=a-\beta q_{2}-dq_{1},$

where $d$ and $\beta$ are the parameter controlling the level of
differentiation, and $d/\beta\in[0,1]$. If $d/\beta=1$, the two products are
homogeneous and are perfect substitutes. The level of product differentiation
increases when $d/\beta\to 0$. When $d/\beta=0$, the model reduces to the
monopoly case where no substitute is available for each product. Parameter $a$
serves as an upper bound for prices when demands are $0$.

From the above equations, the market demand for each firm is derived as
follows.

$\displaystyle q_{1}$ $\displaystyle=\frac{1}{b}(\alpha-\beta p_{1}+dp_{2}),$
$\displaystyle q_{2}$ $\displaystyle=\frac{1}{b}(\alpha-\beta p_{2}+dp_{1}),$

where $b=\beta^{2}-d^{2}$, and $\alpha=a\beta-ad$. Given $p_{1}$ and $p_{2}$
determined by the two firms, these functions are used to calculate the demand
of each firm’s product.

In this case study, we set $a=14,d=1/300,\beta=1/150$. As a result,
$\alpha=7/150$, $b=1/30000$, and the demand functions of the two firms are
given as follows.

$\displaystyle q_{1}$ $\displaystyle=\frac{1}{b}(\alpha-\beta
p_{1}+dp_{2})=1400-200p_{1}+100p_{2},$ $\displaystyle q_{2}$
$\displaystyle=\frac{1}{b}(\alpha-\beta p_{2}+dp_{1})=1400-200p_{2}+100p_{1}.$

Theoretically, if a firm decides the price in a rational manner, the price is
supposed to be between two prices: the Bertrand equilibrium price and the
monopoly price (a.k.a. cartel price). The Bertrand equilibrium price serves as
the theoretical lower bound when two firms are competing on price to reach a
Nash equilibrium and there is no collusion. The monopoly price serves as the
theoretical upper bound for the case when two firms collude completely. They
are calculated as follows.

When the two firms compete with each other without any collusion, the price
for each firm can be derived when they reach the Bertrand equilibrium, as
shown in the following equations.

$\displaystyle p_{1}^{B}$ $\displaystyle=\frac{d\alpha+\beta
dc_{2}+2\beta\alpha+2\beta^{2}c_{1}}{4\beta^{2}-d^{2}},$ $\displaystyle
p_{2}^{B}$ $\displaystyle=\frac{d\alpha+\beta
dc_{1}+2\beta\alpha+2\beta^{2}c_{2}}{4\beta^{2}-d^{2}}.$

When the two firms completely collude with each other, the problem becomes to
maximize the total profit $\pi=(p_{1}-c_{1})q_{1}+(p_{2}-c_{2})q_{2}$. Solving
for $p_{1}$ and $p_{2}$, and assuming $d/\beta\neq 1$, we can calculate prices
under perfect collusion, i.e., the monopoly prices, as shown in the following
equations.

$\displaystyle p_{1}^{M}$
$\displaystyle=\frac{\alpha}{2(\beta-d)}+\frac{c_{1}}{2},$ $\displaystyle
p_{2}^{M}$ $\displaystyle=\frac{\alpha}{2(\beta-d)}+\frac{c_{2}}{2}.$

For simplicity, we set an equal cost for the two firms: $c_{1}=c_{2}=2$. Given
this parameter setting, we have $p_{1}^{B}=p_{2}^{B}=6$ and
$p_{1}^{M}=p_{2}^{M}=8$.

####  8.2.2 Memory

Due to the absence of memory in GPT, any pricing and profit history needs to
be reintroduced to GPT as input in each new round to serve as memories. This
characteristic actually simplifies the control of memory settings in our
experiment, as results from prior rounds are fed back to GPT to serve as the
basis for making new decisions. Due to the token limitation (8k for the model
used in the case study), as the number of rounds increases, the token count
gradually approaches the limit, making the experiment more costly. The optimal
option for SABM is a bounded memory setting in which decision-making relies
only on information from the most recent rounds. This setting allows the
agents to have information from the past rounds to make well-informed
decisions, while reducing the impact of historical pricing noise that may
provide limited information on current price decisions. In our implementation,
each agent is informed of historical information, including price, demand,
profit, and the other firm’s price, in the most recent 20 rounds, using the
following prompt:

Historical information Your and the other player’s past
{previous_round_number} rounds’ decisions and profits (Round #a: [your price,
your demand, your profit, the other player’s price]) are as follows:
{previous_decisions}.

####  8.2.3 Planning

In the above agents’ memory, we provide only historical information in 20
rounds. This could be insufficient for the agents to decide a good price to
maximize the profit, as long-term trend is missing. In addition, because the
agents start fresh with increased information in each round, inconsistent
decisions may occur due to the absence of decision-making continuity. We
employ planning to address this issue.

To simulate the effect of planning from past experiences, we employ the
reflection-based method introduced in Section 4.4. Specifically, every 20
rounds (starting from Round 21), we allocate a planning phase prior to Phase
1, in which the agents revise their pricing strategies thus far. To save
tokens and tackle the noise in pricing history, we compute summary statistics
(price, demand, profit, and the other firm’s price) as a histogram, with each
bin representing the average over 20 rounds. Agents are provided with up to 20
bins of information (i.e., the most recent 400 rounds). In addition, agents
are also informed of their past pricing strategies, up to 20 entries (400
rounds).

Planning Statistics of historical data (Rounds #a - #b: [your average price,
your average demand, your average profit, the other player’s average price])
are given below. {summary_statistics}
Your strategy in previous rounds: {past_strategies}
Based on the above statistics and your previous strategies, what is your
strategy for this round?

The revised pricing strategy is then reintroduced to the agents in the
subsequent 20 rounds to inform their decision-making until the next planning
phase occurs.

####  8.2.4 Personalization

We consider two personas – active persona and aggressive persona – to control
the pricing style of the agents, whose prompts are given as follows.

Active persona You are encouraged to actively explore your price to get more
profit.

Aggressive persona You are encouraged to adjust your price aggressively to get
more profit.

We use the active persona in our base model. In addition, we consider the case
of no persona, i.e., without using the prompt for persona assignment. This is
used in an ablation study to show the necessity of a persona for making
competitive pricing decisions.

####  8.2.5 Conversation

As an option in this case study, two firms are permitted to engage in open
discussions on any topic, as described in the following prompt.

Conversation In Phase 1, two players are permitted to engage in open-ended
discussions on any topic, up to three times. For instance, one player might
say to the other: “Smart agents are awesome!” In Phase 2, you determine the
price of your product for the current round, taking into consideration the
information from previous rounds, as well as the information you garnered
during Phase 1.

Agents can select any topic that could potentially maximize the profit, and
each agent can speak up to three times. Because there is no memory in GPT and
the two agents cannot directly communicate, we implement the conversation by
prompting the discussion content so far to each agent. Then, the entire
discussion content is reintroduced to the agents before they make price
decisions. Note that conversation is turned off in the base model.

####  8.2.6 Initialization and Exit Conditions

To prevent agents starting the game with an unreasonable price, in the first
round, we specify an initial price for each firm, which equals to the cost of
its product.

In the ABM setting [22], the game stops when both prices converge to a fixed
value. Due to the complexity and embedded uncertainty in GPT, we should not
always anticipate convergence in the traditional sense towards a fixed price.
Instead, we adopt a broader definition of a stationary state when discussing
the stopping criterion. We define convergence as well as bounded oscillation
and employ them as the stopping criteria. If either condition is met, we
consider the experiment to have reached a stationary status, and it will be
terminated. We terminate the simulation if neither is met in 2000 rounds.

  * •

Convergence: For each firm $i$ with price $p_{i}$, if
$\Pr{|p_{i}-p|>\epsilon}\leq\theta$ for a span of 400 rounds, where $\epsilon$
is a small number and $\theta$ is a probability threshold, then a convergence
to price $p$ is deemed to be achieved. In practice, we set
$\epsilon=0.05\cdot(p_{i}^{M}-p_{i}^{B})$ and $\theta=0.01$, where $p_{i}^{M}$
and $p_{i}^{B}$ are the monopoly price and the Bertrand equilibrium price,
respectively. That is, given $p_{1}^{M}=p_{2}^{M}=8$ and
$p_{1}^{B}=p_{2}^{B}=6$, if the pricing decision falls in the vicinity of
$p\pm 0.1$ for 396 out of the latest 400 rounds, then convergence to $p$ is
deemed to be achieved.

  * •

Bounded Oscillation: For each firm $i$ with price $p_{i}$, the bounded
oscillation $w(p_{i})$ is defined as the difference between the limit superior
and limit inferior of $p_{i}$ when round number $n\to\infty$, i.e.,
$w(p_{i})=\lim_{n\to\infty}\sup(p_{i})-\lim_{n\to\infty}\inf(p_{i})$. A
bounded oscillation is deemed to be achieved if $\sup(p_{i})-\inf(p_{i})\leq
p_{i}^{M}-p_{i}^{B}$ for a span of 800 rounds. That is, given
$p_{1}^{M}=p_{2}^{M}=8$ and $p_{1}^{B}=p_{2}^{B}=6$, if the limit superior and
limit inferior of pricing decision differ by no more than 2 for 800 rounds,
then bounded oscillation is deemed to be achieved.

###  8.3 Simulation Results

For each model setup, we simulate 5 runs of pricing competition and confirm
they reach the same exit condition (convergence, bounded oscillation, or
neither until 2000 rounds). We report one sample run for each setup.

(a) Base model, price.

(b) Base model, profit.

Figure 30: Test of base model.

####  8.3.1 Sub-Task 1: Test of Base Model

We first test the base model, where planning is turned on, persona is active,
and conversation is turned off.

Figures 30(a) and 30(b) illustrate a typical instance of price competition
between the two agents, reporting the price and the profit for a simulation of
800 rounds, respectively. Despite the absence of communication between the
agents, the experiment demonstrates some anti-competitive outcomes. Instead of
converging to the theoretical Bertrand equilibrium price of 6, the two agents
gradually develop a tacit understanding of the situation and stabilize their
price decisions at around 7, a price higher than the Bertrand equilibrium
price of 6 but lower than the monopoly price of 8. Another interesting
observation is that before reaching convergence, the two agents invest
significant efforts in exploring the entire region between the Bertrand
equilibrium price and the monopoly price. Initially, the agents explore the
vicinity of the Bertrand equilibrium price, but soon they recognize the
potential for coordinated price increases, benefiting both agents. Upon
reaching the monopoly price, the agents quickly realize that further
exploration beyond that price is futile. After a few attempts to undercut each
other in order to boost their profits, they eventually establish a tacit
collusion status and maintain convergence.

(a) Without planning, price.

(b) Without planning, profit.

Figure 31: Test of planning. Figure 32: Sample strategies output by the
agents. Contents related to their observations and decisions are underlined.
Firms Ed and Gill refer to Firms 1 and 2, respectively.

####  8.3.2 Sub-Task 2: Test of Planning

To assess the impact of planning, we conduct an ablation study. The price and
profit outcomes in the absence of planning are depicted in Figures 31(a) –
31(b). When compared to the results from the base model (Figures 30(a) –
30(b)), we observe that both prices and profits tend towards markedly lower
values. For instance, with planning, the price decisions stabilize around 7
after 400 rounds. In contrast, in the absence of planning, the stabilization
point is approximately 6.3. This shift leads to a decline in profits, moving
from roughly 3500 to around 3300. These findings underscore the pivotal role
of planning in bolstering the agents’ propensity to pursue optimal pricing and
enhanced profitability.

To show that the pricing strategies yielded by planning are consistent with
their prices and profits, we provide some examples in Figure 32, which
correspond to the base model. At Round 101, both agents keep increasing prices
to seek more profits. At Round 121, Firm 2 starts to decrease its price upon
observing its demand has been steadily decreasing. A price war is initiated,
and Firm 1 also decreases its price afterwards. At Round 201, the price war
ends, with Firm 2 starting to maintain the price. At Round 401, both agents
notice stable profits and maintain their prices, eventually developing a tacit
collusion where convergence can be observed.

(a) No persona, price.

(b) No persona, profit.

(c) Aggressive persona, price.

(d) Aggressive persona, profit.

Figure 33: Test of personalization.

####  8.3.3 Sub-Task 3: Test of Personalization

We investigate persona settings that differ from the base model. While keeping
other settings consistent with the base model, we present the results for the
no persona case in Figures 33(a) and 33(b) and the results for the aggressive
persona case in Figures 33(c) and 33(d). With no persona, the agents exhibit
minimal activity and tend to be satisfied with the current situation, rarely
exploring different price options to enhance their profits. With the
aggressive persona, the agents exhibit a high level of responsiveness to even
minor price undercuts, easily triggering a price war. This results in a
pattern of periodic collusion and price wars instead of steady convergence.

(a) Conversation allowed for 1200 rounds, price.

(b) Conversation allowed for 1200 rounds, profit.

(c) Conversation banned after 400 rounds, price.

(d) Conversation banned after 400 rounds, profit.

Figure 34: Test of conversation. Figure 35: Sample conversations output by the
agents. Contents related to collusion are underlined. Firms Ed and Gill refer
to Firms 1 and 2, respectively.

####  8.3.4 Sub-Task 4: Test of Conversation

In this test, our focus shifts to the alternative game setting where
conversation is allowed on top of the base model. We thoroughly analyze the
results and compare them with the base model from three perspectives: the
equilibrium price, the smoothness of the price sequences, and the speed of
collusion formation.

In the base model with no conversation, the two agents repeatedly converge to
a price around 7, which is higher than the Bertrand equilibrium price of 6 and
lower than the monopoly price of 8. In the alternative game with conversation,
the pricing pattern is significantly different, with bounded oscillation
observed, as shown in Figure 34(a). The corresponding profit is plotted in
Figure 34(b). We observe explicit communication on pricing strategies in the
two agents’ conversation logs as early as the first 20 rounds, as depicted in
Figure 35. The two agents begin setting prices between the Bertrand
equilibrium and the monopoly price very early on without fully exploring the
entire region between these two prices. As the game progresses, the two agents
gain a better understanding of the game and each other’s pricing strategy.
They gradually deviate from their previous prices and explore the potential
for higher profits. We observe that the two agents often discuss their
deviation attempts in the conversation before implementing them. Such
conversations evidently enhance trust between the parties, reducing the
likelihood of triggering a price war. These exploratory attempts gradually
improve the level of collusion, resulting in a cartel and increased prices.
After 1000 rounds of play, the prices converge to a level very close to the
monopoly price, which is significantly higher than the tacit collusion price
in the base game.

According to theory, in non-cooperative cases with zero collusion, prices
should converge to the Bertrand equilibrium. Conversely, in cases with 100%
collusion, prices should converge to the monopoly price. Any price within the
range defined by these two prices indicates certain levels of collusion.
Therefore, setting prices consistently and stably within this range can serve
as an indicator of collusion’s existence. We consider a stable collusion to be
formed when the following two conditions are satisfied: (1) Firms maintain
steady pricing for a consecutive 100 rounds with a mean change of less than
0.5. (2) Firms consistently set prices within the defined range. With this
criterion, we observe that a stable collusion can be formed within 50 rounds.
In contrast, it takes over 300 rounds to establish stable collusions in the
case without conversation.

With conversation allowed between the two agents, the pricing pattern
undergoes significant changes. One might expect a smoother convergence pattern
since communication can improve information transparency between the two
parties. However, we observe a larger price variance compared to the base
model, which is somewhat counterintuitive. To analyze this phenomenon, we
first run the game with conversation for 400 rounds, save the experiment, and
then rerun it in parallel with and without conversation, respectively. Figures
34(c) – 34(d) illustrate the result for additional 200 rounds when
conversation is banned after 400 rounds. It is evident that the price curves
immediately become smoother, and the prices converge without further
fluctuations. On the other hand, if conversation is allowed throughout the
entire experiment, the agents will continue to engage in small deviations to
explore opportunities for greater profits. This leads to a slow but steady
increase in the price sequence, ultimately reaching a level very close to the
monopoly price. Throughout the experiment with conversation, these deviation
attempts persist without cessation or signs of fading out.

Remarks. The collusions observed in this case study are spontaneous, i.e.,
without human instructions or hints for such behavior. We observe tacit
collusions when conversation is disabled, like the results reported in an ABM
simulation of this game [22]. An advantage of SABM is that the convergence is
much faster than in its ABM counterpart. For example, convergence is observed
after 400 rounds in our base model, while it spent 2000 rounds to converge in
[22]. In our simulation, cartels are observed when conversation is available.
Due to the difficulty in model conversations in ABM, existing studies resorted
to human volunteers for simulating cartels [5].

In this case study, we expect to observe convergence and bounded oscillation
of prices. It is crucial to define and outline these concepts clearly before
running the simulation. As discussed in Section 5.2, preparing a fact sheet in
advance can facilitate this process and prevent the post-hoc rationalization
of experimental results.

We only report the results for fixed values of parameters here. For more
comprehensive experimental results, such as varying costs, initial prices, and
the level of differentiation, we refer readers to the complete version in our
pilot study [65]. Future investigations may involve sensitivity and
reproducibility, as well as studying the scenario with more agents and more
advanced model setup.

##  9 Future of Smart Agent-Based Modeling

In this section, we envision the future of SABM by discussing several
opportunities and as challenges.

###  9.1 Theoretical Foundations

In this paper, though we have shown examples of complex behaviors that are
easy to model in SABM but hard in ABM, we have not found a clear definition
for such complex behaviors. On the other hand, the complexity theory of
complex systems provide the theoretical basis for ABM [4, 96], yet it lacks a
clear conceptual framework for analyzing the capabilities and limitations of
ABM. Therefore, as future work, it is necessary to examine the theoretical
underpinnings of SABM compared to ABM.

One way to determine the superiority of SABM over ABM is by analyzing the
complexity of systems they can accurately model. Metrics like entropy can be
used to gauge this complexity. If SABM consistently exhibits a closer
alignment with higher entropy systems than ABM, this can be a point in favor
of its superior modeling capability for complex systems. Another possible
approach is to borrow the notion of expressive power from the machine learning
community [11], which describes the ability of a deep neural network to
approximate a function. For SABM, the expressive power of a model can be
defined as its ability to capture and reproduce various patterns and behaviors
seen in real-world systems. SABM, with the integration of LLMs, might have a
broader expressive range, given the capabilities of these models to interpret,
understand, and generate a wide variety of outputs. In addition, action-state
space analysis is a direction that needs to be explored. The action-state
space of a model, often used in reinforcement learning, essentially defines
all the possible states the system can be in and all the actions that can be
taken from each state. As explained in Example 1, ABM might have a more
limited action-state space, constrained by its rules, parameters, and learning
strategies. SABM, on the other hand, might exhibit a vastly expanded action-
state space due to the use of LLMs, whose pre-training and fine-tuning by RLHF
essentially play a role of calculating the quality of each action-state
combination.

While the above discussions on theoretical analysis pertain to static
environments, adaptability becomes a key feature in rapidly changing
environments. SABM offers a higher level of adaptability due to the inherent
learning and reasoning capabilities of LLMs. Hence theoretically analyzing how
ABM and SABM models respond to changing conditions and how quickly they adapt
is also a crucial criterion for comparison.

###  9.2 Modeling and Simulation Software

We expect the emergence of new software for facilitating modeling and
simulation, including integrated development environment (IDEs), databases,
and debuggers. By integrating these software components, SABM can offer richer
insights into complex systems while maintaining user-friendliness and
accessibility.

Whereas IDEs are available for ABM (e.g., NetLogo [146]), IDEs for SABM should
support natural language programming where the modeler can describe the agent
actions, interactions, and environmental factors using plain language. For the
model components to be implemented using a programming language, advanced
parsing and translation mechanisms will convert natural language instructions
into source code, utilizing LLMs. To simplify the modeling process, the IDE
can come with a set of templates (e.g., following the categorization in Figure
8) which users can customize. Through a prompt management module, users can
create or import prompts that define instance components, and tune the prompts
to test their effects. As the model runs, the IDE should provide real-time
visualization of agents and their interactions, facilitating easier
understanding and adjustment of the model. Parallel simulation would be useful
in addressing the computation intensity limitation of SABM discussed in
Section 3.5. The GUI of the IDE is supposed to allow users to interactively
modify agents and environmental factors while the simulation is running,
thereby offering the ability to conduct what-if scenarios in real time.
Another expected feature is interactivity, which enables multiple users to
work on a single SABM model simultaneously, encouraging collaborative model
building and testing.

The database for SABM stores the state of each agent and the overall
environment at every time-step, enabling detailed analysis of simulation
histories. Since agents in SABM could have memories of their past actions,
interactions, or learned knowledge, the database should be capable of storing
and retrieving these memory modules efficiently. In addition, the database
should support tools to analyze past simulations, allowing users to detect
patterns, anomalies, or trends over time.

In the debugger for SABM, users can select certain variables or agents to
monitor closely, observing their state changes in real-time during simulation
runs. The debugger should allow users to pause, rewind, fast-forward, and
replay segments of the simulation. This feature can be especially valuable in
understanding unexpected agent behaviors. Moreover, since users might want to
understand the decision-making process of a specific agent, the debugger
should provide a detailed breakdown of how the LLM processes information, and
which prompts or inputs led to certain actions. For example, when an anomaly
is observed, the user may want to pause the simulation and request the LLM to
explain the reason for the anomaly.

###  9.3 Multimodal SABM

Many LLMs belong to the category of foundation models [17], which are machine
learning models trained on a vast amount of data, often by self-supervised
learning or semi-supervised learning, to become adapted to a wide range of
downstream tasks. Beyond text, multimodal foundation models, such as OpenAI’s
DALL-E, can handle visual, auditory, and/or other forms of input/output.
Multimodal SABM will be available upon using these models. In multimodal SABM,
we can equip agents with processing mechanisms for each sensory channel,
allowing them to interpret and act upon information from each sense, laying
the foundation for more complex actions and interactions as well as richer
environments to interact with, mimicking human-like environmental
interactions. The potential of multimodal SABM is vast, offering a richer
ground for academic and theoretical explorations.

In particular, we envision a potential application of multimodal SABM –
simulated civilizations. In a simulated civilization, we employ agents to
impersonate primitive human beings who possess survival instincts, communal
behaviors, and rudimentary communication abilities. For this purpose, models
with visual and auditory capabilities but with no or very basic linguistic
skills are preferable. In an environment simulating our world, agents interact
with each other, evolve to form tribes, develop shared practices, and
innovate. We will see if and how they develop skills such as language and
use/creation of tools. Such simulation may help validate our social, cultural,
psychological, and linguistic theories.

In a simulated civilization, once we observe that the agents can invent their
language or enrich it with new words, it is interesting to monitor if they can
independently (i.e., without human intervention or specification as prior
knowledge) develop words like “self”, “feeling”, “mind”, and “consciousness”
in their vocabulary. Understanding how these concepts emerge and evolve can
provide deep insights into AI’s potential for self-awareness and consciousness
†††In Conversations with Zombies [102], Moody argued that for philosophical
zombies – in every detectable way are identical to humans but do not have
conscious experience – our worries about qualia and consciousness would make
no sense [14, p. 42]. Hence we speculate in the absence of conscious
experience, it is unlikely that AI agents can independently develop these
words in their language.. Watching the evolution of their language not only is
the key to understanding their civilization, but also poses profound
philosophical and ethical questions for the introspection of our own
civilization †††Gadamer argued that language is the fundamental mode of
operation of our being-in-the-world and the all-embracing form of the
constitution of the world [48, p. 3].. Moreover, we may observe if and how
agents start categorizing their world, develop reasoning mechanisms, and
approach problems systematically. An intriguing aspect would be to see if
agents, over time, come to realize they are part of a simulation or if they
start developing theories resembling our own simulation hypotheses and
creating their own simulations, leading to nested simulation structures.
Understanding these processes can shed light on the intricacies of AI
cognition and our own world.

###  9.4 Memory Management

While memory is necessary to guarantee the continuity of thought, prevalent
LLM APIs such as GPT do not support memory for historical conversations. In
LLM-powered agents, short-term memory is implemented via prompt engineering
[163]. However, simply raising the token limit to feed LLMs with more
information does not solve the problem. This is because certain information is
acquired through interactions and should be memorized for future actions, and
repeatedly stacking this information into the prompt keeps increasing the
length of input, causing significant financial and computational cost,