# Efficient Behavior-consistent Calibration for Multi-agent Market Simulation

Tianlang He<EMAIL_ADDRESS>0000-0002-4939-5993 ,  Keyan Lu
<EMAIL_ADDRESS>The Hong Kong University of Science and TechnologyClear
Water BayHong Kong S.A.R. ,  Chang Xu  ,  Yang Liu  ,  Weiqing Liu  chanx,
yangliu2<EMAIL_ADDRESS>Microsoft Research AsiaBeijingChina ,
S.-H. Gary Chan<EMAIL_ADDRESS>The Hong Kong University of Science and
TechnologyHong Kong S.A.R. and  Jiang Bian<EMAIL_ADDRESS>Microsoft
Research AsiaBeijingChina

###### Abstract.

Order-driven market simulation mimics the trader behaviors to generate order
streams to support interactive studies of financial strategies. In market
simulator, the multi-agent approach is commonly adopted due to its
explainability. Existing multi-agent systems employ heuristic search to
generate order streams, which is inefficient for large-scale simulation.
Furthermore, the search-based behavior calibration often leads to inconsistent
trader actions under the same general market condition, making the simulation
results unstable and difficult to interpret.

We propose CaliSim, the first search-free calibration approach multi-agent
market simulator which achieves large-scale efficiency and behavior
consistency. CaliSim uses meta-learning and devises a surrogate trading system
with a consistency loss function for the reproducibility of order stream and
trader behaviors. Extensive experiments in the market replay and case studies
show that CaliSim achieves state-of-the-art in terms of order stream
reproduction with consistent trader behavior and can capture patterns of real
markets.

Market simulation, multi-agent system optimization, meta-learning

##  1\. Introduction

In order-driven markets, say, the stock market, participants interact and
compete for maximal profits through order-making, during which the order
streams are generated. Usually, the order stream can be characterized by
market states, marked by many crafted economic indicators such as the
unemployment rate. Market simulation is to mimic the mechanism behind order
stream generation, which enables many impactful applications in the financial
industry, such as trading strategy back-testing (harvey2015backtesting, ) and
risk management (gupta2013risk, ).

Literature in market simulation devises agents to generate order streams,
which can be categorized as single and multi-agent market simulations. The
single-agent market simulation (coletta2021towards, ; shi2022state, ) use one
agent to generate orders, representing the whole market. The multi-agent
simulation emulates real exchange environments by employing heterogeneous
agents – order streams are generated over multi-agent interactions. Multi-
agent simulation provides very details of the trading process, and thus, is
more explainable than its counterpart, enabling it easy to interpret the
simulation results.

In a multi-agent simulation system, each agent represents one (or a group of)
market participants. Various types of agents have been devised and
successfully shown the ability to reflect real market participants, though,
these agents require calibrations on their _behavior parameters_ , i.e., the
parameters set for the trading strategy, value assessment method, investment
horizon, and so on. Since the real market evolves over time, one constant set
of behavior parameters cannot always faithfully reflect market situations over
all periods. This calls for efficient and careful calibration methods for
multi-agent simulation systems.

Figure 1. The system overview of CaliSim. During inference, the meta-market
directly generates a set of behavior parameters by analyzing the target order
stream with market state indicators. During model training, we leverage a
surrogate trading system to facilitate order stream reproduction.
Additionally, the meta-market enforces the two consistency rules by a proposed
consistency loss.

One essential goal for the calibration of the multi-agent simulation is to
reproduce the target order stream. Existing works (bai2022efficient, ;
cao2022dslob, ) calibrates behavior parameters by heuristic search: for each
time unit (such as one trading day), they heuristically search for a set of
parameters for the multi-agent system to generate similar order streams to the
target. Although encouraging results have been reported, the search process is
tedious and time-consuming, making them inefficient for large-scale (e.g., the
year-basis) simulations.

Additionally, due to the randomness and dynamics of the multi-agent system, it
is possible that several sets of behavior parameters lead to similar order
streams. Failing to consider such calibration redundancy, the simulations with
these parameters, calibrated by current approaches, can perform differently
with further interventional studies, such as trading strategy back-testing,
making their resultant simulations unstable and arbitrary.

In this paper, we target efficient multi-agent system calibration for order
stream reproduction in one shot, based on the observation that the calibration
experiences share commonalities over time. Specifically, since behavior
parameters contribute to order stream generation, we can capture the behavior
information from the target order stream for calibration, retrospectively.

To tackle the calibration redundancy, we argue that the calibration for
behavior parameters should observe the patterns in the real market. In
particular, the overall behaviors of the market participants are temporally
dependent. Thus, while targeting order stream reproduction, we introduce two
additional behavior consistency patterns to reduce the calibration redundancy:

  * •

_Temporal consistency:_ Without a striking exogenous intervention (e.g., the
Black Swan), the overall behaviors of the market participants should vary
smoothly and continuously;

  * •

_Market state consistency:_ Similar market states usually indicate close
trading environments. Thus, the overall behaviors of the market participants
should be similar under close market states.

Based on the observations, we propose CaliSim, an efficient and behavior
consistent _cali_ bration for multi-agent market _sim_ ulation and overview
the system in Figure 1. During inference, CaliSim employs a meta-market that
directly generates behavior parameters by analyzing the target order stream
with market state indicators, which avoids the tedious parameter searching
process. During model training, it leverages a surrogate trading system for
order stream reproduction and a consistency loss function to comply with the
consistency rules aforementioned.

CaliSim is an end-to-end system and considers generalization ability in its
design. The main novelties of our system are listed as follows:

  * •

_Meta-market design to achieve generalized calibration:_ Although market
states characterize participant behaviors, they usually cannot directly
determine behaviors due to their macro properties. We consider this by
treating the market states as mechanisms, that is, under different market
states, we use distinct ways to analyze order streams. This leads to our meta-
design of the model: the order stream determines behavior parameters, while
the market state decides the way it analyzes. Such a meta-design explores the
mechanisms behind the input distribution, which enables a generalized
calibration;

  * •

_Surrogate trading system for reparameterization:_ The multi-agent system is
dynamic, complex, and thus, not differentiable. This poses a big challenge for
the end-to-end meta-market to reproduce target order streams. We tackle this
by re-parameterizing the multi-agent system. Specifically, we observe that the
behavior parameters contribute and correlate to some important features in the
generated order stream. Thus, we learn a surrogate trading system to capture
the correlation between the behavior parameter and the features, constructing
a substitution of the multi-agent system to facilitate the meta-market
learning.

  * •

_Consistency loss to achieve consistent calibration:_ We propose a consistency
loss function to achieve the two behavior consistency patterns. For temporal
consistency, we treat the units as time series and minimize the behavior
variations. To enforce the market state consistency – participants perform
similarly under similar market states, given other factors unchanged – we
devise a contrastive hypothesis scheme: hypothesize real markets with various
market states and permutes the behavior parameters contrastively.

By accumulating episteme from past calibration experiences, CaliSim enables
efficient search-free calibration, offering a stable simulation that reflects
patterns in real markets. Additionally, CaliSim considers and interpolates the
market state, which not only calibrates but also makes it possible to scenario
hypothesis. This further extends its usage to more applications, such as
stress scenario testing.

We have implemented and conducted extensive experiments to verify CaliSim on
the orderbook-level data from the A-share market over a year, noting that the
existing market simulation approaches mostly experiment for up to a few days.
The experimental results show that CaliSim can achieve similar reproduction
results, compared to existing calibration works, while observing the
consistency rules. We also conduct case studies to show that CaliSim can
capture some patterns in the real market.

The paper’s contributions are summarized as follows:

  * •

To the best of our knowledge, CaliSim is the first searching-free calibration
system for multi-agent market simulation;

  * •

We first show how to leverage exogenous information, outside the orderbook, to
calibrate the multi-agent market simulation system;

  * •

We have conducted large-scale (on a year basis) experiments to verify CaliSim
and firstly illustrates scenario hypothesis based on the system.

The remainder of this paper is organized as follows. We first introduce
preliminaries in Section 2, followed by system design in Section 3, where the
three novel modules: surrogate trading system (Section 3.2), meta-market
(Section 3.1), and consistency loss (Section 3.3) are included. Then, we show
the illustrative experimental results in Section 4 and finally conclude in
Section 5. We discuss related works in Section 6.

##  2\. Preliminary

###  2.1. Limit Order Book

Figure 2. Pictorical demonstration of the limit order book.

More than half of the markets in today’s competitive and relentlessly fast-
paced financial world now use a limit order book (LOB) mechanism to facilitate
trading (gould2013limit, ). As shown in Figure 2, the limit order book works
like a shelf placing and matching orders (the blocks), where market
participants interact. In an orderbook, the tick size is the smallest
permissible price interval between orders, i.e., the minimal interval between
two grids on the shelf; the lot size refers to the smallest amount of the
asset that can be traded within an order book, i.e., the unit size of the
block. The gap between the two best quotes (best bid and ask price) is called
the spread, whose middle is mid-price. More inclusive descriptions of LOB can
be found in (gould2013limit, ).

The three most basic operations that LOB provides for the participants are
limit order placement, market order placement, and order cancellation. A limit
order is mainly marked by five characteristics: order ID, timestamp, direction
(ask/bid), price, and size. Placing a limit order to LOB can be an analogy to
putting a colored block onto the shelf in Figure 2. The LOB will automatically
match limit orders whose prices make a deal. The market order can be viewed as
a special limit order, whose price is settled to be the best quote of its
adversary side. After order placement, the traders can cancel their orders
that have not been executed by LOB.

LOB contains the very microscope market information of the order stream, i.e.,
the order time series arrival at the LOB. Order stream over LOB often
generates prominent features or microscopic dynamics which are called stylized
facts, such as volatility clustering and gain-loss asymmetry. Readers
interested may refer to (vyetrenko2020get, ) and references therein.

###  2.2. Heterogeneous Multi-agent Simulation

Heterogeneous multi-agent simulation is a computational model which can
simulate the actions and interactions of individuals and organizations, in
complex and realistic ways. Even with simple agents, it can exhibit complex
behavior patterns and provide valuable information about the dynamics of the
real-world system which they emulate (iori2012agent, ). Such a simulation
system usually consists of an order book and a group of agents.

We follow the taxonomy in (chiarella2009impact, ) and introduce some classic
types of agents:

  * •

The zero-intelligence agent, i.e., noise trader, usually makes orders whose
price and size conform to some invariant statistical distributions. This type
of agent is commonly considered to model some irrationalities in the market.
Also, the zero-intelligence agent contributes to some realistic market
dynamics in the simulation system;

  * •

The chartist estimates the price of investment objects by historical data from
the order book; thus, many machine learning-based agents belong to this group;

  * •

The value-based agent, or the fundamentalist, assesses an asset’s value by a
common belief among the markets; in the implementation, these agents usually
estimate prices according to an exogenously provided value, which is named as
the fundamental value.

There are also many other types of agents, such as the market maker who makes
orders around the mid-price to provide liquidity. Overall, each type of agent
reflects a part of real traders from reality, enabling the multi-agent
simulation system to replay the markets.

Nevertheless, these agents need some configurations to trade. For instance,
the agents usually need to cancel their order if it is not executed after a
period, i.e., investment horizon, which requires calibration in advance. Also,
the order price and size are sometimes correlated; this can be modeled by many
utility functions, such as CARA (babcock1993risk, ) and CRRA
(chiarella2006asset, ), where we need to designate different risk aversions to
different agents. Besides, different types of agents may require distinct
parameters. For example, the noise trader requires assumptions for its price
distribution; the chartist needs to settle its value assessment strategy on
assets.

The heterogeneous multi-agent simulation system can be either continuous or
discrete. This mainly concerns the agents’ order placement interval. Since it
is not the focus of the paper, we focus on discrete multi-agent simulation,
where the orders are aggregated and executed by time slot. But our approach
makes no assumption on the discrete simulation, it can easily extend to
continuous systems.

##  3\. CaliSim: Consistent Calibration for Multi-agent Simulation

###  3.1. Meta-market

Figure 3. Pictorial demonstration of the mechanism that multi-agent market
simulation tries to model: various factors impact participants’ behavior
(model calibration); the behavior dominates the real market to generate order
stream (simulation execution).

Figure 3 shows the mechanism that market simulation tries to model: various
factors impact the market participants’ behavior which dominates the real
market to generate order stream. In the process, calibration methods model the
impact mechanism to produce behavior parameters; and executing the multi-agent
simulation system based on the behavior parameters embodies the later part.
Our goal is to build a calibration model, named meta-market, to produce
behavior parameters.

As mentioned in Section 2.2, many types of agents are designed to equip multi-
agent simulation systems with different model assumptions. Here, we denote
$M_{h}$ as the multi-agent model with an assumption $h$

(1) $M_{h}(b)=s,$

where the model $M_{h}$ generate order stream $s$ based on _behavior
parameters_ $b$. Thus, the meta-market should provide the multi-agent system
$M_{h}$ with behavior parameters so that the multi-agent model $M_{h}$ mimics
well to reality, even upon further interventional studies, i.e., high
fidelity.

In real markets, there are many factors that impact market behaviors, Among
the factors, some of them are explicitly identified in economics and
characterized by macro indices (i.e., market state indicators), such as CPI
and PPI (AKShareZhaiQuanShuJuAKShare, ). Besides, more implicit factors that
directly impact market behaviors exist without being named.

To enable search-free calibration, the meta-market needs to model the impact
mechanism shown in Figure 3. The explicit factors can be manually selected,
though, the implicit factors are innumerable, and thus, hard to model. Here,
we observe that implicit factors determine market behaviors that dominate the
target order streams. Thus, we extract features from the target order stream
to represent the implicit factors.

The implicit features are extracted from the target order stream in binary
ways:

  * •

Market factors dominate a price estimation upon a market, which leads to a
price trend in order stream. Thus, we extract mid-price series from the order
stream to facilitate agents with value assessments, by serving as the
fundamental price (lespagnol2018trading, ) for agents who possess value-based
trading strategies;

  * •

Order stream is much more informative than merely price trend; for example,
order volume can reflect the liquidity of a market. To sufficiently exploit
information from target order streams, we build a data-driven implicit feature
extractor for implicit factor representation

(2) $u=p_{\theta_{1}}(s),$

where the extractor $p_{\theta_{1}}$ extracts order stream $s$ to produce
implicit feature $u$.

Given the explicit and implicit features, we can build an impact mechanism
model

(3) $K\left(u,x\right)=b,$

where it produces behavior parameter vector $b$ based on implicit and explicit
features $u$ and $x$.

Usually, the two factors contribute differently to market behaviors in real
markets – explicit factors tend to collectively characterize participant
behaviors over a period, while implicit factors impact the markets more
directly and promptly. For example, although a market tends to be passive
during a period of a high unemployment rate, some temporal market
irrationalities can still lead to a price increase in stock.

Figure 4. Illustration of meta-market design. Meta-market takes order stream
and market state (explicit features) as input and produces behavior vector. To
model the different impacts of implicit and explicit factors on market
behaviors, meta-market adopts the meta-design in the impact mechanism model.

We involve such an observation in building the impact mechanism model: the
behavior vector is decided by implicit features while the explicit feature
determines the way it decides. Such a design resembles an analyst – it
analyzes and speculates the market behaviors based on the more direct
(implicit) factors, while the market states (explicit factors) decide the way
they think. This leads to our meta-design of the mechanism model

(4) $K\left(u,x\right)=q_{\theta_{2}}(x),$

where we estimate behavior via a behavior estimator $q_{\theta_{2}}$; and the
model hyperparameter of the behavior estimator is based on a market state
analyzer

(5) $\theta_{2}=g_{\omega}(u).$

We finally overview the meta-market in Figure 4 (mid-price extraction is
omitted for simplicity). Meta-market first employs an implicit feature
extractor to extract the implicit feature from order stream. Then, a behavior
estimator $q_{\theta_{2}}$ produces a behavior vector based on the implicit
feature, during which a market state analyzer $g_{\omega}$ considers the
explicit feature to decide the model assumption, i.e., hyperparameters, of the
behavior estimator. The market state analyzer and behavior estimator, overall,
constitute the impact mechanism as shown in Figure 3.

###  3.2. Surrogate Trading System

Since the meta-market is a data-driven system, we need objective functions to
supervise the model estimating behavior vectors with high fidelity according
to the real order streams. However, it is impossible to know the ground truth
optimal behavior $b^{*}$ of each time unit. Existing works regard a similar
reproduction as high fidelity – for the $t^{th}$ time unit, if the estimated
behavior vector $\hat{b_{t}}$ is close to $b_{t}^{*}$ by a gap $\epsilon$,
i.e.,

(6) $\left\lVert\hat{b_{t}}-b_{t}^{*}\right\rVert_{n}<\epsilon,$

the generated order stream should be close to the ground truth order stream
$s_{t}$

(7) $\left\lVert
M_{h}\left(\hat{b_{t}}\right)-s_{t}\right\rVert_{n}<k\epsilon,$

where $M_{h}$ denotes the multi-agent system and $k>0$. Therefore, the current
calibration approaches find the behavior parameter $\hat{b_{t}}$ by heuristic
searching.

The search process is tedious, painful, and ineffective, leaving order stream
information not fully utilized. Thus, we propose an end-to-end meta-market
system to directly analyze order stream data and estimate behavior parameters,
in a search-free manner. However, the multi-agent system is complex, dynamic,
and thus, not differentiable. This poses a big challenge to optimizing the
meta-market by the reproduction objective in Equation 7.

We tackle this through the observation that behavior parameters of multi-agent
systems usually contribute to some statistical features in the generated order
stream. For example, more chartists in the market often lead to a short-tailed
return distribution over mid-price. This enables us to re-parameterize the
multi-agent system by a surrogate trading system, so that

(8) $\min_{\gamma\in Y}\ \left\lVert
sur_{\gamma}(b)-Q\left(M_{h}(b)\right)\right\rVert_{n},$

where the $sur$ denotes the surrogate model, parameterized by $\gamma$, and
the $Q$ function calculates those features that the simulation users consider
to be important. In this way, we re-parameterize the non-differentiable multi-
agent system by the surrogate trading system and use it to optimize the meta-
market $K$ via a reproduction loss

(9) $L_{repr}=\left\lVert
sur_{\gamma}\left(\hat{b_{t}}\right)-Q\left(s_{t}\right)\right\rVert_{n},$

where $\hat{b_{t}}=K(u_{t},x_{t})$.

###  3.3. Consistency Loss

As mentioned in Section 1, due to the randomness and dynamics of the multi-
agent simulation system, more than one set of behavior parameters can
contribute to similar order streams. which means reproduction is not enough to
indicate high fidelity. In order words, Equation 7 is not sufficient to
indicate Equation 6.

We select the candidates by introducing some patterns in the real market,
which potentially pushes the simulation one step further toward high fidelity.
Specifically, we observe that the behaviors are usually dependent over time:

  * •

Without a striking intervention, the overall behaviors of the market usually
vary smoothly and continuously, which we call _temporal consistency_ ;

  * •

Similar market states usually indicate similar trading environments. Thus, the
overall behaviors of the market participants should be similar under close
market states, which we called _market state consistency_.

The end-to-end structure of the meta-market makes it easy to enforce temporal
consistency: over a period of $T$ time units, the meta-market should find
behavior vectors so that

(10) $\min_{\theta_{1},\omega}\
\sum_{t=1}^{T-1}\left\lVert\hat{b_{t}}-\hat{b}_{t+1}\right\rVert_{n},$

where the estimation is produced by the meta-market
$\hat{b_{t}}=K(u_{t},x_{t})$, parameterized by $\theta_{1}$ and $\omega$. This
leads to our proposed temporal consistency loss

(11) $L_{temp}=\sum_{t=1}^{T-1}\left\lVert
K(u_{t},x_{t})-K(u_{t+1},x_{t+1})\right\rVert_{n}.$

The market state consistency is more tricky to consider because it requires
other factors to remain unchanged, which is hard to determine and find such
perfect matches. We consider this issue by a contrastive hypothesis scheme –
for each time unit, we fabricate its market states by hypothesis and permute
the resultant behaviors over the hypothesized data. This leads to our
objective function on the market state consistency over the $t^{th}unit$,
i.e.,

(12) $\min_{\omega}\
sim\left(u_{t}^{(\alpha)},u_{t}^{(\beta)}\right)\left\lVert
K\left(u_{t}^{(\alpha)},x_{t}\right)-K\left(u_{t}^{(\beta)},x_{t}\right)\right\rVert_{n},$

where the $u_{t}^{(\alpha)}$ and $u_{t}^{(\beta)}$ are two fabricated market
states on the $t^{th}$ time unit and the $sim(\cdot)$ denotes the similarity
of two market states. Note that we should only optimize the market state
analyzer (as shown in Figure 4) using the objective because this helps with
the generalization upon the model level.

We enforce the objective in Equation 12 by a market state consistency loss
function based on triplet loss

(13)
$L_{stat}=\max\left\\{\left\lVert\hat{b}_{t}-\hat{b}_{t}^{(\alpha)}\right\rVert_{n}-\left\lVert\hat{b}_{t}-\hat{b}_{t}^{(\beta)}\right\rVert_{n}+\sigma,0\right\\},$

where
$sim\left(u_{t},u_{t}^{(\alpha)}\right)>sim\left(u_{t},u_{t}^{(\beta)}\right)$
and the $\sigma$ denotes a contrastive margin. In this way, the market state
consistency loss function encourages the behavior estimations to be similar
under similar market states with other factors unchanged.

Besides, such a hypothesis scheme not only calibrates but also enables us to
conduct a market hypothesis – the market state consistency loss function
interpolates market states of the model inputs, which allows us to modify the
market states on real trading days to see how the order streams change
accordingly.

Overall, the meta-market is trained by a composite loss function

(14) $L=L_{repr}+w_{t}L_{temp}+w_{s}L_{stat},$

where the two coefficients $w_{t}$ and $w_{s}$ weigh the two consistency loss.

###  3.4. Multi-agent System Building

We build a discrete multi-agent system to verify the performance of CaliSim.
In the system, the orders are made and executed collectively by time slot –
each agent and the matching algorithm of the orderbook only wake up once in
one slot. During a wake-up, an agent place one order and, if needed, any
instructions on order cancellation, and the orderbook executes all agent
requests at the end of each time slot. Note that we do not consider short-
selling in this experiment. We implement the system in Python.

We designate the time slot to be one second, that is, the orderbook makes
14400 executions on agent requests in a trading day with four exchange hours.
In a time slot, each agent decides to make an order or not. This models the
order-placing interval of a trader. As it is not the focus of the paper, we
set the wake-up of agents to be independent and probabilistic: the wake-up
probability of each agent in each time slot is $1\%$. To guarantee the
heterogeneity of the agents, we involve 500 agents in the system. Addtionally,
each agent has an investment horizon: the agent will cancel its orders if the
orders’ lifecycles exceed the agent’s horizon (not being executed by the
orderbook). To unify the system, we use time reference $\tau$ to serve as the
reference horizon of the system.

We build the agents based on the work in (chiarella2009impact, ) – the price
and size of the order are correlated by CARA utility function
(babcock1993risk, ) which denotes that the consequent high risk makes the high
profit less attractive, as the expected profit increases. Based on the
understanding, the size of the order is determined by its price

(15) $\pi(P)=\frac{\log(\hat{P}_{t+\tau}/P)}{\alpha V_{t}P},$

where $\pi$ is the expected asset holding quantity, $\hat{P}_{t+\tau}$ is the
agent’s estimated price at time $t+\tau$, and $V_{t}$ is the variance of
historical mid-price. We follow the paper and sample the price $P$ from a
uniform distribution whose range is bounded by the agent’s account (leverage
is not considered in its paper).

Each agent is a composite of three agent types – fundamentalist, chartist, and
noise trader. Agents mainly differentiate each other by proportions of the
agent types, parameterized as $g_{f}$, $g_{c}$, and $g_{n}$, respectively. The
estimated price of each agent is calculated as

(16)
$P_{t+\tau}=\frac{g_{f}P_{f}^{(t)}+g_{c}P_{c}^{(t)}+g_{n}P_{n}^{(t)}}{g_{f}+g_{c}+g_{n}},$

where $P_{f}^{(t)}$ denotes the estimated price by a fundamentalist at time
$t$ and so do $P_{c}^{(i)}$ and $P_{n}^{(i)}$ for the chartist and the noise
trader. The horizon of the $i^{th}$ agent is

(17) $\tau^{(i)}=\tau\frac{1+g_{f}^{(i)}}{1+g_{c}^{(i)}},$

and its risk aversion, similarly, is determined as

(18) $\alpha^{(i)}=\alpha\frac{1+g_{f}^{(i)}}{1+g_{c}^{(i)}},$

where the $\alpha$ is the reference risk aversion of the system.

We initialize each agent’s account by sampling from a uniform distribution,
whose range is related to the asset price (so that agents can afford to bid
and ask). The different types of agents mainly differentiate by price
estimation methods:

  * •

The fundamentalist estimates asset price by an exogenous reference denoting a
common sense about the true value of the asset, i.e., the fundamental value.
In the system, we sample from the mid-price of the simulated order stream as
the fundamental value and empirically select the sample interval as ten
minutes;

  * •

The chartist produces price estimation by historical mid-price in the market.
In our experiment, we employ the chartist linear regression to estimate asset
price, where the chartist looks back upon the historical data length of the
agent’s horizon;

  * •

The noise trader determines asset price by probability. In our setting, we use
Gaussian distribution, that is, the estimated price conforms to
$\mathcal{N}(P_{t},\sigma^{2})$ where $p_{t}$ is the current mid-price at time
$t$ and the standard deviation is set to be $1\%$ of the open price of each
trading day;

  * •

Institutional investor estimates asset prices in the same way as regular one.
But its account is abundant with more initial cash and holding of shares; and
thus, it is likely to place orders in large sizes. In deployment, we double
the account distribution range against the regular agent to initialize initial
holdings of institutional investors.

_Behavior parameters in out experiment:_ We sample $g_{f}$, $g_{c}$, and
$g_{n}$ from Laplacian distribution, parameterized by $\delta_{f}$,
$\delta_{c}$ and $\delta_{n}$. In the experiments, we demonstrate CaliSim by
optimizing these parameters and investment horizon unit $\tau$ and the
probability of an agent being an institutional investor.

##  4\. Illustrative Experimental Results

###  4.1. Experimental Settings

Figure 5. Learning process of the surrogate trading system.

Figure 6. Learning process of the meta-market with reconstruction and temporal
consistency loss.

Figure 7. Distribution of behavior variation on the training set.

We verify CaliSim by two experiments: market replay and case study on the
market state. In the market replay, we evaluate CaliSim by order stream
reproduction, measured by the _reconstruction error_ , and compare _behavior
variation_ to show that CaliSim observe the temporal consistency rule. In the
case study, we justify that the meta-market, trained by the market state
consistency loss function, is able to capture some real market patterns.

We experiment with orderbook-level data of a stock (WindCode: 000001) from the
A-share market in the whole year of 2020. We treat a trading day as a unit
during which the behavior parameters remain unchanged. In detail, we use data
from the first nine months for model training and the left three months for
test. To represent the market state, we select five well-studied indicators:
consumer price index (CPI), producer price index (PPI), purchasing manager’s
index (PMI) (available in (AKShareZhaiQuanShuJuAKShare, )), market trend, and
market noise – among them, CPI, PPI and PMI are monthly updated; and, we
calculate the market trend based on average true range (ATR) and market noise
using efficiency ratio (ER) over a month.

We compare CaliSim with the following schemes:

  * •

_Random search (RandSearch)_ (cao2022dslob, ) arbitrarily samples behaviors in
parameter space and selects the best parameter combination that reproduces the
target order stream. In the implementation, we try ten combinations for each
trading day;

  * •

_Bayesian optimization (BayesianOPT)_ (bai2022efficient, ) searches for
behavior parameters in each step by referring to its historical steps. In the
implementation, we conduct Bayesian optimization based on the Gaussian process
and search in ten steps for each trading day.

We train the surrogate model by collecting a dataset using the built multi-
agent simulation system – for each trading day, we randomly select ten sets of
behavior parameters to generate order streams. The selected behavior
parameters, as well as the fundamental value of the day, will be the surrogate
model input. For the output of the surrogate system, we extract features (as
the feature extractor $Q$ in Section 3.2) from the order stream in four
aspects:

  * •

_Mid-price return distribution_ reflects price trend in micro view. The return
at time $t$ is calculated as

(19) $r_{t}=\log\left(\frac{P_{t}}{P_{t-\Delta t}}\right),$

where use minutely return and extract from return distribution of these
features: the gain-loss ratio, kurtosis, and the ratio of zero return;

  * •

_Volatility clustering_ reflects how intensive mid-price volatility clusters
in time. It is usually calculated by autoregression function
(vyetrenko2020get, ): $corr(r^{2}_{t},r^{2}_{t+n\Delta t})$. Here, $\Delta t$
is one minute and we use $n=1,2,3$ and the average of the ten values as the
surrogate model output.

  * •

_Order size distribution_ is the distribution of the limit order sizes within
a trading day. In the experiment, we consider order sizes that are less than
100 lots and use, the ratio of orders whose sizes are less than 1, 5, 10, and
50, as the surrogate model output;

  * •

_Order price from current mid-price_ denotes the price gap between the price
of a new order and the current mid-price. We evaluate its distribution from
one to ten tick sizes. and extract the ratio of orders whose price deviates
not more than one and five ticks from its current mid-price.

We compare and evaluate our approach through the following metrics:

  * •

_Reconstruction error:_ We sum the normalized MSE of the four mentioned
aspects to evaluate the reproduction quality between the generated and target
order stream, where we use the z-score for normalization;

  * •

_Behavior variation:_ We sum the MSE of all behaviors over two consecutive
trading days and refer it as behavior variation. We also use the z-score to
balance the different behavior parameters.

Since the behavior parameters are temporally correlated, we extract the
implicit feature by LSTM from order streams of a month.To facilitate model
learning, we extract some features, like the label of the surrogate model,
from the order stream of each trading day as the input of the meta-market. In
model building, we set the LSTM to two layers with two hidden layers. For the
market state analyzer $g_{\omega}$, we use two fully connected (FC) layers
with neurons of 200 and 100, respectively. For the behavior estimator, we use
one FC layer to connect to transform the analyzed feature to behavior
parameters. To balance the input of the surrogate model, we first reduce the
dimension of the fundamental value from 24 to 4 by one FC layer with sizes of
50 and employ three fully-connected layers with 50, 100, and 50 to
reparameterize the multi-agent system. We use ReLU to provide non-linearity
and employ Adam Optimizer for training. Finally, we empirically set
$w_{t}=0.1$ and $w_{s}=1$ in Equation 14.

Figure 8. Distribution of behavior variation on the test set.

Figure 9. Comparison on order stream reproduction in the training set.

Figure 10. Comparison on order stream reproduction in the training set.

Figure 11. Reconstruction loss with and without market state consistency loss function.  Figure 12. Correlation between behavior parameters and market trend.  Table 1. Correlation analysis between market state indicators and behavior parameters  | RandSearch | BayesianOPT | CaliSim
---|---|---|---
CPI | 0.0764 | 0.0447 | 0.2555
PPI | 0.0790 | 0.0513 | 0.2595
PMI | 0.0494 | 0.0549 | 0.2634
Market Trend | 0.0466 | 0.0921 | 0.3266
Market Noise | 0.0122 | 0.0601 | 0.3102

###  4.2. Illustrative Results

####  4.2.1. System setup: the learning process of meta-market

We first show the learning process of the surrogate trading system in Figure
7. The figure shows that the loss of the training curve decreases with
training epochs. The loss on the validation set descends at the epoch of ten
and flats off after that. The plot demonstrates a common pattern of learning
curves about training deep learning models. This shows that reparameterizing
the multi-agent market simulation using feature extraction is feasible, which
validates our design of the surrogate trading system. In the following, we
select the model at the epoch of ten as the surrogate trading system.

We demonstrate the training process of the meta-market employing
reconstruction (by the surrogate trading system) and temporal consistency loss
in Figure 7. It shows that both reconstruction error and behavior variation
are reduced in the training process, where both metrics decrease drastically
in the first fifteen epochs and the reduction slows down after that. This
verifies the possibility of optimizing both objectives concurrently – there
exist several sets of behavior parameters that contribute to similar order
streams, and the proposed two loss functions can optimize the two aspects by
selecting parameter sets with temporal consistency.

####  4.2.2. Market replay: evaluation on order stream reproduction and
temporal consistency of the behavior parameters.

We compare our approach with existing arts, as introduced in Section 4.1, and
show their distribution of behavior variations over the first nine months
(training set) in Figure 7. From the histogram, most behavior variations
(between trading days) of the comparison schemes reside in the range from two
to four. Such large variations do not reflect the market regularity that the
overall participant behaviors vary continuously. Notably, the behavior
variation of RandSearch is slightly larger than that of BayesianOPT; this is
because RandSearch introduces more randomness during its parameter searching.
In comparison, CaliSim consider the temporal consistency of the behaviors and
employ temporal consistency loss to train the meta-market. Thus, most behavior
variations of CaliSim, from the figure, fall into the range from zero to two,
which observes the market regularity in calibration.

To verify the generality of CaliSim, we further compare behavior variations
over the last three months (the test dataset) and show the results in Figure
10. The figure shows that the behavior variation distribution of both
comparison schemes do not change against the distribution in the training set.
This is because they conduct searching for each trading day independently.
Undergoing an insignificant behavior variation increase, though, CaliSim still
performs much better than random search and Bayesian optimization. This
verifies that CaliSim has learned to conduct consistent calibration by
considering the temporal consistency loss.

Figure 10 compares the reconstruction error by cumulative distribution
function (CDF) over calibration on trading days in the training dataset. From
the figure, CaliSim performs similarly to Bayesian optimization to achieve low
reconstruction error, while random search shows unsatisfactory results. This
is because both CaliSim and BayesianOPT involve objectives to reduce the
error, while the parameter searching of RandSearch is pretty random. Even
though, both the comparison schemes require a search process with ten
iterations, while CaliSim searches parameters in one shot. This verifies the
efficiency of CaliSim.

We further verify the reconstruction error of the comparison schemes on the
test set in Figure 10. It shows that the performance of RandSearch and
BayesianOPT does not change much since they search parameters independent of
trading days. In the figure, even CaliSim performs similarly to BayesianOPT in
terms of reconstruction error, though, CaliSim is free from the tedious
process of parameter searching and can find parameters that observe
consistency rules (as demonstrated in Figure 10). This shows the superiority
of CaliSim over its comparison schemes, where it successfully accumulates
calibration knowledge from historical data.

We conduct an ablation study on the market state consistency loss function and
compare CaliSim with or without the loss function in terms of reconstruction
error, as shown in Figure 11. The figure shows that considering the market
state facilitates calibration to reduce reconstruction error in both training
and test sets. Notably, both schemes experience slight error increases from
training to test data, though, the error increase of CaliSim with the loss
function is smaller and more negligible, compared to its counterpart. This
verifies that the market state is informative of calibration; and, CaliSim can
successfully explore such information and leverage it to enhance its
generalization ability.

####  4.2.3. Case study on the market state: CaliSim can capture real market
patterns

To show that CaliSim is informative of market state by employing the market
state consistency loss, we calculate the correlation coefficients between the
market state indicators and the output behavior parameters and average the
absolute value of the correlation coefficients in Table 1. From the table, the
produced behavior parameters of CaliSim are much more correlated to the market
state indicators than the comparison schemes. This shows the ability of
CaliSim to explore the market state information, as well as the effectiveness
of the market state consistency loss function.

We conduct a case study on market trend and show its correlation coefficients
with the calibrated parameters in Figure 12: when the market trend becomes
significant, CaliSim calibrates by increasing the ratio of chartists and
institutional investors while reducing the number of noise traders and the
investment horizon of the whole market. This reflects the market patterns –
the number of chartists increases will increase since their trading strategies
fit markets with significant trends; naturally, the number of noise traders
decreases with clear signals from the trends. Meanwhile, institutions tend to
enter stock markets when the market shows significant trends, or, sometimes,
the influences are mutual. Also, under such scenarios, participants who make
profits are more likely to take profit, as well as the liquidation, leading to
a low investment horizon over the whole market.

##  5\. Conclusion

Multi-agent market simulation generates order streams through agent
interactions. To enable the simulation that reflects reality, the agents need
careful calibration of their behaviors. Existing calibration approaches
heuristically search behavior parameters for reproducing the target order
stream. However, failing to accumulate calibration episteme, existing
approaches undergo the tedious searching process, independently, for each
trading day, making their applications hard to scale up. Furthermore, due to
the randomness and dynamics of the system, it is possible that more than one
sets of parameters (redundancy) contribute to similar order streams – these
parameters can perform differently with further interventional studies, such
as trading strategy back-testing, making simulation unstable and random.

In this paper, we aim to build a calibration system without parameter
searching. Since behavior parameters dominate the simulation system, resulting
in the generated order streams, we can explore behavior information from the
target order stream for reproduction, which captures the calibration knowledge
between trading days. To tackle the parameter redundancy, we argue that
calibration should observe some patterns of the real market and propose two
consistency rules: temporal and market state consistency. To enforce the
observations, we propose CaliSim, a behavior-consistent calibration system for
multi-agent simulation. CaliSim is the first searching-free calibration system
that directly analyzes order stream and market states to produce behavior
parameters in an end-to-end manner. To achieve calibration generality to
unseen trading days, CaliSim is designed based on meta-learning and trained by
a proposed surrogate trading system. To observe the two consistency rules, we
propose a consistency loss function. We have conducted large-scale experiments
(the largest, to the best of our knowledge) to verify our approach by market
replay and case studies. The experimental results of market replay show that
CaliSim can achieve similar order stream reproduction while observing the
consistency rules. The case studies show that CaliSim can reflect some
patterns of the real market.

##  6\. Related Work

With the advancement in data mining (dogan2021machine, ) and storage
technologies (chen2022tvconv, ), market analysis using orderbook-level data
has flourished in the field of financial technology, such as return prediction
(hou2021stock, ; hou2022multi, ; shah2019stock, ) and LOB recreation
(shi2021lob, ; shi2021limit, ). Among them, market simulation
(assefa2020generating, ) leverages the orderbook information to explore its
generation mechanism, which enables many financial applications such as
trading strategy-back testing (dowd2012back, ; gort2022deep, ;
vezeris2020optimization, ), risk management (gupta2013risk, ; leo2019machine,
), and stress testing (reinders2023finance, ).

Market simulation mainly leverages proxies, representing participants, to
interact with the orderbook, which can be categorized as the single and multi-
agent market simulation by proxy duties. The single-agent market simulation
(coletta2021towards, ; shi2022state, ; li2020generating, ;
coletta2022learning, ) models an aggregated behavior via one consolidated
proxy to make orders, which interacts with the orderbook, the so-called “world
agent”; being conceptually straightforward, though, the model suffers poor
explainability. Multi-agent market simulation (liu2021finrl, ; byrd2019abides,
; storchan2021learning, ; karpe2020multi, ) represents each agent as one (or a
group of) market participants with parameterized behaviors; each agent strives
to maximize its proprietary profits, where order streams are generated over
their interactions. Multi-agent simulation provides very details of trading,
such as agent account variations, which are essential for users to understand
the simulation results. Nevertheless, the agents of the multi-agent market
simulation require calibration on hyperparameters, or behavior parameters, to
reproduce target order streams.

Most approaches calibrate the multi-agent simulation system through “try and
see”: try some behavior parameters with the multi-agent system and compare the
output order stream in terms of reproduction to target. Works in
(chiarella2009impact, ; cao2022dslob, ) empirically set behavior parameters by
random search; the approach in (storchan2021learning, ) utilizes grid search
to reduce the randomness during the parameter search, compromising the
accuracy of order stream reproduction. However, these calibration approaches
depend on a vast number of trials for each trading day, which is tedious and
labor-intensive. Work in (bai2022efficient, ) leverages Bayesian optimization
to expedite the parameter searching process, where each parameter set is
dependent on its historical search results. Even though, the search process
for each calibration unit is independent, which is still inefficient for
large-scale applications. Furthermore, the randomness and dynamics of the
multi-agent simulation render calibration redundancy where the simulations may
perform differently under interventional studies, and thus, produces unstable
and random results. In comparison, CaliSim explores the dependency between
trading days and tackles the calibration redundancy by patterns from the real
markets, which calibrates in one shot and produces behavior parameters that
reflect real market patterns.

## References

  * (1) AKShare AKShare 1.8.62 document.  https://www.akshare.xyz/data/bond/bond.html#id37.
  * (2) Assefa, S. A., Dervovic, D., Mahfouz, M., Tillman, R. E., Reddy, P., and Veloso, M. Generating synthetic data in finance: opportunities, challenges and pitfalls.  In Proceedings of the First ACM International Conference on AI in Finance (2020), pp. 1–8.
  * (3) Babcock, B. A., Choi, E. K., and Feinerman, E. Risk and probability premiums for cara utility functions.  Journal of Agricultural and Resource Economics (1993), 17–24.
  * (4) Bai, Y., Lam, H., Balch, T., and Vyetrenko, S. Efficient calibration of multi-agent simulation models from output series with bayesian optimization.  In Proceedings of the Third ACM International Conference on AI in Finance (2022), pp. 437–445.
  * (5) Byrd, D., Hybinette, M., and Balch, T. H. Abides: Towards high-fidelity market simulation for ai research.  arXiv preprint arXiv:1904.12066 (2019).
  * (6) Cao, D., El-Laham, Y., Trinh, L., Vyetrenko, S., and Liu, Y. Dslob: A synthetic limit order book dataset for benchmarking forecasting algorithms under distributional shift.  arXiv preprint arXiv:2211.11513 (2022).
  * (7) Chen, J., He, T., Zhuo, W., Ma, L., Ha, S., and Chan, S.-H. G. Tvconv: Efficient translation variant convolution for layout-aware visual processing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022), pp. 12548–12558.
  * (8) Chiarella, C., Dieci, R., and Gardini, L. Asset price and wealth dynamics in a financial market with heterogeneous agents.  Journal of Economic Dynamics and Control 30, 9-10 (2006), 1755–1786.
  * (9) Chiarella, C., Iori, G., and Perelló, J. The impact of heterogeneous trading rules on the limit order book and order flows.  Journal of Economic Dynamics and Control 33, 3 (2009), 525–537.
  * (10) Coletta, A., Moulin, A., Vyetrenko, S., and Balch, T. Learning to simulate realistic limit order book markets from data as a world agent.  In Proceedings of the Third ACM International Conference on AI in Finance (2022), pp. 428–436.
  * (11) Coletta, A., Prata, M., Conti, M., Mercanti, E., Bartolini, N., Moulin, A., Vyetrenko, S., and Balch, T. Towards realistic market simulations: a generative adversarial networks approach.  In Proceedings of the Second ACM International Conference on AI in Finance (2021), pp. 1–9.
  * (12) Dogan, A., and Birant, D. Machine learning and data mining in manufacturing.  Expert Systems with Applications 166 (2021), 114060.
  * (13) Dowd, K. Back-testing market risk models.  Encyclopedia of Financial Models (2012).
  * (14) Gort, B. J. D., Liu, X.-Y., Sun, X., Gao, J., Chen, S., and Wang, C. D. Deep reinforcement learning for cryptocurrency trading: Practical approach to address backtest overfitting.  arXiv preprint arXiv:2209.05559 (2022).
  * (15) Gould, M. D., Porter, M. A., Williams, S., McDonald, M., Fenn, D. J., and Howison, S. D. Limit order books.  Quantitative Finance 13, 11 (2013), 1709–1742.
  * (16) Gupta, A. Risk management and simulation.  CRC Press, 2013.
  * (17) Harvey, C. R., and Liu, Y. Backtesting.  The Journal of Portfolio Management 42, 1 (2015), 13–28.
  * (18) Hou, M., Xu, C., Li, Z., Liu, Y., Liu, W., Chen, E., and Bian, J. Multi-granularity residual learning with confidence estimation for time series prediction.  In Proceedings of the ACM Web Conference 2022 (2022), pp. 112–121.
  * (19) Hou, M., Xu, C., Liu, Y., Liu, W., Bian, J., Wu, L., Li, Z., Chen, E., and Liu, T.-Y. Stock trend prediction with multi-granularity data: A contrastive learning approach with adaptive fusion.  In Proceedings of the 30th ACM International Conference on Information & Knowledge Management (2021), pp. 700–709.
  * (20) Iori, G., and Porter, J. Agent-based modelling for financial markets.
  * (21) Karpe, M., Fang, J., Ma, Z., and Wang, C. Multi-agent reinforcement learning in a realistic limit order book market simulation.  In Proceedings of the First ACM International Conference on AI in Finance (2020), pp. 1–7.
  * (22) Leo, M., Sharma, S., and Maddulety, K. Machine learning in banking risk management: A literature review.  Risks 7, 1 (2019), 29.
  * (23) Lespagnol, V., and Rouchier, J. Trading volume and price distortion: an agent-based model with heterogenous knowledge of fundamentals.  Computational Economics 51 (2018), 991–1020.
  * (24) Li, J., Wang, X., Lin, Y., Sinha, A., and Wellman, M. Generating realistic stock market order streams.  In Proceedings of the AAAI Conference on Artificial Intelligence (2020), vol. 34, pp. 727–734.
  * (25) Liu, X.-Y., Rui, J., Gao, J., Yang, L., Yang, H., Wang, Z., Wang, C. D., and Guo, J. Finrl-meta: A universe of near-real market environments for data-driven deep reinforcement learning in quantitative finance.  arXiv preprint arXiv:2112.06753 (2021).
  * (26) Reinders, H. J., Schoenmaker, D., and van Dijk, M. A finance approach to climate stress testing.  Journal of International Money and Finance 131 (2023), 102797\.
  * (27) Shah, D., Isah, H., and Zulkernine, F. Stock market analysis: A review and taxonomy of prediction techniques.  International Journal of Financial Studies 7, 2 (2019), 26.
  * (28) Shi, Z., and Cartlidge, J. The limit order book recreation model (lobrm): An extended analysis.  In Machine Learning and Knowledge Discovery in Databases. Applied Data Science Track: European Conference, ECML PKDD 2021, Bilbao, Spain, September 13–17, 2021, Proceedings, Part IV 21 (2021), Springer, pp. 204–220.
  * (29) Shi, Z., and Cartlidge, J. State dependent parallel neural hawkes process for limit order book event stream prediction and simulation.  In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (2022), pp. 1607–1615.
  * (30) Shi, Z., Chen, Y., and Cartlidge, J. The lob recreation model: Predicting the limit order book from taq history using an ordinary differential equation recurrent neural network.  In Proceedings of the AAAI Conference on Artificial Intelligence (2021), no. 1, pp. 548–556.
  * (31) Storchan, V., Vyetrenko, S., and Balch, T. Learning who is in the market from time series: market participant discovery through adversarial calibration of multi-agent simulators.  arXiv preprint arXiv:2108.00664 (2021).
  * (32) Vezeris, D. T., Schinas, C., Kyrgos, T. S., Bizergianidou, V., and Karkanis, I. Optimization of backtesting techniques in automated high frequency trading systems using the d-backtest ps method.  Computational Economics 56 (2020), 975–1054.
  * (33) Vyetrenko, S., Byrd, D., Petosa, N., Mahfouz, M., Dervovic, D., Veloso, M., and Balch, T. Get real: Realism metrics for robust limit order book market simulations.  In Proceedings of the First ACM International Conference on AI in Finance (2020), pp. 1–8.