# Alleviating Non-identifiability: a High-fidelity Calibration Objective for
Financial Market Simulation with Multivariate Time Series Data

Chenkai Wang1, Junji Ren2, Peng Yang1,2111Corresponding author
1Department of Statistics and Data Science,
Southern university of Science and Technology, China
2Guangdong Provincial Key Laboratory of Brain-Inspired Intelligent Computation
Department of Computer Science and Engineering
Southern university of Science and Technology, China
$*$Corresponding author
Emails<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The non-identifiability issue has been frequently reported in the social
simulation works, where different parameters of an agent-based simulation
model yield indistinguishable simulated time series data under certain
discrepancy metrics. This issue largely undermines the simulation fidelity yet
lacks dedicated investigations. This paper theoretically analyzes that
incorporating multiple time series data features in the model calibration
phase can alleviate the non-identifiability exponentially with the increasing
number of features. To implement this theoretical finding, a maximization-
based aggregation function is applied to existing discrepancy metrics to form
a new calibration objective function. For verification, the financial market
simulation, a typical and complex social simulation task, is considered.
Empirical studies on both synthetic and real market data witness the
significant improvements in alleviating the non-identifiability with much
higher simulation fidelity of the chosen agent-based simulation model.
Importantly, as a model-agnostic method, it achieves the first successful
simulation of the high-frequency market at seconds level. Hence, this work is
expected to provide not only a rigorous understanding of non-identifiability
in social simulation, but a high-fidelity calibration objective function for
financial market simulations.

##  1 Introduction

Social simulation plays a pivotal role of understanding the complex social
systems in a way of revealing their endogenous complexity [1]. The core is to
construct a parameterized simulation model that can generate the simulated
time series data resembling the observed states over time of the social
system. By properly modeling the endogenous components of the social system as
multiple interacting agents, i.e., the Agent-based Modeling (ABM) techniques
[2], various “what-if” tests can be conducted [3]. That is, by intervening
certain agents of the simulation model, the corresponding endogeneity of the
real social system is revealed by observing how the evolution of the
simulation model is influenced by the intervention.

The success of a social simulation largely lies with the model fidelity. Note
that, the what-if tests are often counterfactual, i.e., they do not happen in
real social systems and thus there are no ground-truth for validation. To what
extent should we believe the results of what-if tests? In social systems, due
to the uncertainty of human activities, it is often impractical to derive
trustworthy simulation models mathematically or physically. Usually, this
issue is addressed in a data-driven manner via model calibration [4].

The basic idea is that, if a simulation model $M(\omega)$ can generate
simulated data $\mathbf{X}_{T}$ resembling the observed data
$\hat{\mathbf{X}}_{T}$ of the targeted real system within the discrete time
interval $T=[t_{s},t_{s}+1,\cdots,t_{e}]$, it is considered that the
underlying probabilistic distribution of the real system for generating
$\hat{\mathbf{X}}_{T}$ is learned by $M(\omega)$. Here, $\hat{\mathbf{X}}_{T}$
denotes the time series $\\{\hat{\mathbf{X}}_{t}\\}^{t=t_{e}}_{t_{s}}$ for
short. Hence, while using $M(\omega)$ for what-if test, by keeping the non-
intervened agents of $M(\omega)$ unchanged, the simulation results of how the
intervened agent influences the system are considered trustworthy. To obtain
such $M(\omega)$, the discrepancy between $\mathbf{X}_{T}$ and
$\hat{\mathbf{X}}_{T}$ is measured by various metrics $D$ [5]. The smaller $D$
is, the closer $M(\omega)$ approximates the distribution of data generation
behind the real social system. By fixing the ABM-based model structure $M$,
calibrating the parameter $\omega$ to fit a given observed data sequence
$\hat{\mathbf{X}}_{T}$ is often modeled as an optimization problem [5]:

$\min_{\omega\in\Omega}D(\hat{\mathbf{X}}_{T},M(\omega)=\mathbf{X}_{T})$ (1)

In practice, the above data-driven calibration methods often suffer from the
so-called non-identifiability issue [6, 7]. That is, there are multiple
parameters in the parameter space $\Omega$ enjoy the same discrepancy value,
given the same $\hat{\mathbf{X}}_{T}$. However, those parameters highly likely
lead to different distributions of data generation, due to the nonlinear model
structure $M(\omega)$. In other words, the “ground-truth” parameter, deciding
the “ground-truth” data generation distribution, cannot be effectively
identified from the other parameters in those sets. Consequently, the what-if
tests on the resultant model of a randomly picked non-identifiable parameter
therefrom are less trustworthy.

Despite that the non-identifiability is frequently mentioned in the
literature, there is neither a formal definition of it nor a rigorous way to
alleviate it. This paper first defines the non-identifiability issue in the
form of multivariate time-series data. It is then discussed that the non-
identifiability arises from that the previously calibrated time series data
$\hat{\mathbf{X}}_{T}$ is often univariate and is insufficient to describe the
uniqueness of a social system. For simplicity and applicability, this work
restricts the scenarios to the financial market simulation, as it is a
representative social simulation and has been active for over 30 years [8].
Like other complex social systems, the real-world financial markets will
output multivariate states data over time, e.g., price, volume, bid/ask
directions, order arrival time, and other hand-crafted features [9]. However,
existing works merely consider the observed price data for calibration,
resulting in wide criticism for their low simulation fidelity [10], especially
in intraday simulations [11, 6].

This paper alleviates the non-identifiability issue by incorporating more data
features into the calibration process, each of which represents a univariate
time series data. We mathematically show that the non-identifiability can be
reduced exponentially with the number of features. To this end, the
multivariate time series is fed into existing discrepancy metrics in a way of
multiple univariate time series. That is, the data associated with each
feature is calibrated individually. This results in multiple individual
calibration tasks with respect to the multiple features, respectively. Note
that, the non-identifiable set of the multivariate time series calibration
task should be non-identifiable across all the individual tasks. Specifically,
only the ground-truth parameter locates simultaneously at the global optimum
of all parameter spaces of those individual tasks. To implement it, we propose
to search in the intersection of all those individual spaces and implement it
via a maximization function aggregating all those individual calibration
tasks. Such an implementation enjoys a high scalability that one can
incorporate as many as possible data features to alleviate the non-
identifiability.

Extensive empirical studies have been conducted to show that the proposed
method can significantly alleviate the non-identifiability issue of financial
market simulation on 10 synthetic data and 1 real data from Shenzhen Stock
Exchange of China. Such advantages are verified robust over 6 commonly seen
financial market data features. Besides, this paper successfully reports the
high-fidelity simulation results of the high-frequency market at seconds
level, which is the first time for the ABM-agnostic calibration methods.

The rest of this paper is organized as follows. Section 2 describes the
background of financial market simulation and related works. Section 3
theoretically discuss the alleviation of non-identifiability. Section 4
reports the empirical studies in details. Section 5 concludes this work.

##  2 Background

###  2.1 ABM-based Financial Market Simulation

In social simulations, Agent-based Modeling (ABM) based models are dominantly
preferred than the black-box deep learning models due to their
interpretability of how the data is generated [12]. ABM constructs a
collection of agents mimicking various social activities and interacting
together to simulate the whole social system [12]. In financial market
simulation, the agents are designed to model various trading strategies of
human [13] (illustrated in Figure 1), e.g., fundamentalists, momentum traders,
high-frequency traders, market makers, and arbitrage traders. Recent financial
market simulation works favor the Preis-Golke-Paul-Schneid (PGPS) model, which
consists of two general types of agents, i.e., the liquidity providers and the
liquidity takers [14].

Figure 1: An illustration of the ABM-based financial market simulator with
various agents interacting with the simulated exchange.

Regardless of the differences among agents, the ABM-based financial market
simulations follow the same workflow. That is, at the beginning of the
simulation, the parameterized agents are initialized with some parameters,
which determine how exactly those agents submit orders under their own
simulated strategies. Each submitted order usually consists of the price,
volume, direction, and arrival time. The exchange is simulated as a Limited
Order Book (LOB) that maintains the existing untraded orders submitted by the
agents [15].

Figure 2: A LOB organizes all the untraded orders in terms of directions,
prices, arrival time and volume.

Those untraded orders are categorized into two directions (bid or ask) and
sorted by their prices. For the orders with the same price, they are again
sorted by the arrival time. Each agent continuously observes the current LOB
and submit its order at will. If a new order matches a price in the opposite
direction of the LOB, it will be traded immediately, and the matched orders
will be removed from the LOB [16]. In existing financial market simulations,
the mid-price (see Figure 2) is mostly considered as the observable time
series data for calibration, as it implies the potential movement of the
trading prices and is important to many what-if tests, e.g., the causes of
flash crashes [17, 18].

###  2.2 Calibration of Financial Market Simulation

To ensure the fidelity of a financial market simulation, the model often needs
careful calibration of the agents’ parameters, so that the simulated data
resembles the observed time series from the real market. The general form of
calibration has been given in Eq. (1).

In recent years, the study of the discrepancy metrics $D$ has attracted
increasing research attentions, as the distance measure between time series
remains unsolved and impacts significantly on the calibration performance.
Methods of Simulated Moments [19] are a typical method that first transforms
the time series into several statistical moments, and then calculates the
weighted average between the moments of observed data and simulated data. The
information criterion based objective functions are also studied [20, 21].
These methods abstract the temporal information of the original time series
via various techniques like histograms and measure the distances between the
abstracted temporal vectors. The non-parametric Kolmogorov-Smirnov test is
also introduced as $D$ by measuring the distance between the probabilistic
distributions of the observed data and the simulated data [7]. Others employ
the tool of Bayesian theory to estimate the likelihood [22] or posterior
distribution [23] of the observed data.

Despite of the progresses of exploiting various metrics for calibrating the
financial market simulation, existing works frequently encounter the so-called
non-identifiability issue, resulting in poor simulation fidelity [24]. From
our viewpoint, one major reason is that existing works merely use the mid-
price data to calibrate the model, ignoring that the real financial market
continuously output the data with multiple features. Although the mid-price
might be important to the market participants, the other features describe the
market in more details. Unfortunately, the above mentioned calibration
objective functions are all designed for univariate time series data (for mid-
price). Consequently, existing works are ill-equipped to calibrate with
multivariate time series data. Yet, it is still unclear how exactly it
benefits from calibrating more features.

##  3 A High-fidelity Calibration Objective

This section first formally defines the non-identifiability, then
theoretically analyzes how it benefits from calibrating with multivariate
data, and lastly proposes a new calibration objective function using multiple
features.

###  3.1 The Definition of Non-identifiability

As existing discrepancy metrics $D$ are designed for univariate data, we re-
denote the calibrated data as $\hat{\mathbf{X}}_{T}^{k}$ with a superscript
$k$ to emphasize that it is a 1-dimensional time series describing some $k$-th
feature of the real system. In financial markets, this feature can be either
price, volume, bid/ask directions, order arrival time, or any other hand-
crafted features [9]. Correspondingly, the simulated data generated by
$M(\omega)$ is re-denoted as $\mathbf{X}^{k}_{T}$. Then the original
calibration task can be re-written as
$\min_{\omega}D(\hat{\mathbf{X}}_{T}^{k},M(\omega)=\mathbf{X}^{k}_{T})$, and
the non-identifiability issue is defined as follows.

Let $\Omega^{D,k}$ be the parameter space of this calibration task. There
exists some region $\mathcal{S}^{D,k}\subseteq\Omega^{D,k}$ that any two
different parameters $\omega_{1},\omega_{2}\in\mathcal{S}^{D,k}$ satisfy
$D(\hat{\mathbf{X}}_{T}^{k},M(\omega_{1}))=D(\hat{\mathbf{X}}_{T}^{k},M(\omega_{2}))$.
Such region $\mathcal{S}^{D,k}$ is called the non-identifiability set and is
non-empty if the ground-truth parameter $\omega^{*}$ is assumed to be included
in $\Omega^{D,k}$. Then the non-identifiability is defined as the probability
of uniformly randomly sampling a parameter $\omega$ from $\Omega^{D,k}$ while
falling into $\mathcal{S}^{D,k}$, denoted as
$P(\omega\in\mathcal{S}^{D,k}|\hat{\mathbf{X}}_{T}^{k})$. Let $|\mathcal{S}|$
denote the cardinality of a set $\mathcal{S}$, it gives

$P(\omega\in\mathcal{S}^{D,k}|\hat{\mathbf{X}}_{T}^{k})=\frac{|\mathcal{S}^{D,k}|}{|\Omega^{D,k}|}$
(2)

Therefore, the rank of any $\omega_{1},\omega_{2}\in\mathcal{S}^{D,k}$ cannot
be effectively identified by the objective $D$ given the data
$\hat{\mathbf{X}}_{T}^{k}$, leading to a flat landscape of the parameter space
therein and the search-based calibration process easily gets stuck in
$\mathcal{S}^{D,k}$. Note that, $\mathcal{S}^{D,k}$ can involve arbitrary
parameters in $\Omega^{D,k}$, including the “ground-truth” parameter
$\omega^{*}$. Hence, if the calibration process converges to
$\mathcal{S}^{D,k}$, one has to randomly pick one $\omega\in\mathcal{S}^{D,k}$
for further what-if simulation. Unfortunately, the underlying data generation
distribution decided by any $\omega\in\mathcal{S}^{D,k}$ is less likely to be
the same with $\omega^{*}$, given the nonlinear nature of the simulation model
$M(\omega)$. Consequently, the what-if test on the randomly picked $\omega$ is
less trustworthy.

###  3.2 Alleviating the Non-identifiability with Multiple Features

In this section, it is shown that the non-identifiability issue can be
alleviated exponentially with $K>1$ features of the observed time series data.

Suppose the observed time series generated by the real social system is
multivariate containing $K$ features, i.e.,
$\hat{\mathbf{X}}_{T}=\\{\hat{\mathbf{X}}_{T}^{k}\\}_{k=1}^{K}$, which is true
for the financial markets. And the simulation model $M(\omega)$ can also
generate a multivariate simulated data
$\mathbf{X}_{T}=\\{\mathbf{X}_{T}^{k}\\}_{k=1}^{K}$ with the same $K$
features. Given that existing discrepancy metrics $D$ only computes for two
univariate time series, there are in total $K$ individual calibration tasks:
$\min_{w}D(\hat{\mathbf{X}}_{T}^{k},M(w)=\mathbf{X}_{T}^{k}),k=1,\cdots,K.$

Note that, given any two parameters $\omega_{1},\omega_{2}\in\Omega^{D,k}$,
only they are simultaneously non-identifiable in all $K$ individual
calibration tasks, i.e.,
$\omega_{1},\omega_{2}\in\\{\mathcal{S}^{D,k}\\}_{k=1}^{K}$, are they non-
identifiable with the general calibration task with respect to
$\hat{\mathbf{X}}_{T}$. Otherwise, their discrepancy to $\hat{\mathbf{X}}_{T}$
can be distinguishable in at least one feature and thus are identifiable. That
is to say, only the intersection of $\\{\mathcal{S}^{D,k}\\}_{k=1}^{K}$ can be
regarded as the non-identifiable set in the multivariate time series setting.
Hence, the non-identifiability in the multivariate time series setting can be
calculated as:

$P(\omega\in\\{\mathcal{S}^{D,k}\\}_{k=1}^{K}|\hat{\mathbf{X}}_{T})=\frac{|\cap_{k=1}^{K}\mathcal{S}^{D,k}|}{|\Omega^{D}|}$
(3)

It is easy to obtain that
$\frac{|\cap_{k=1}^{K}\mathcal{S}^{D,k}|}{|\Omega^{D}|}\leq\frac{|\mathcal{S}^{D,k}|}{|\Omega^{D,k}|},k=1,..,K$.
Hence, we have:

$P(\omega\in\\{\mathcal{S}^{D,k}\\}_{k=1}^{K}|\hat{\mathbf{X}}_{T})\leq
P(\omega\in\mathcal{S}^{D,k}|\hat{\mathbf{X}}_{T}^{k})$ (4)

In Eq.(3) and Eq.(4), the equality holds only if the $K$ non-identifiable sets
are fully overlapped. This can be caused by two reasons. First, the $K$
features are fully dependent that the time series data of one feature can be
deduced from other features. This suggests that the selected features should
be as diverse as possible. Second, the parameter spaces of all the other $K$
individual tasks are full of non-identifiable sets, i.e.,
$\mathcal{S}^{D,k}=\Omega^{D,k},k=1,2,...,K$, which is highly unlikely with
existing discrepancy metrics $D$. Therefore, with proper choice of the
features and $D$, the non-identifiability of calibrating the multivariate time
series $\hat{\mathbf{X}}_{T}$ is smaller than that of the univariate
$\hat{\mathbf{X}}_{T}^{k}$. In other words, calibrating multiple features can
alleviate the non-identifiability issue. Next, we estimate how fast it
decreases with $K$ as follows.

For clarity, let $A_{k}$ denote the event of a uniformly randomly sampled
parameter falling into the non-identifiable set of the $k$-th individual
calibration task. The probability of happening $A_{k}$ is
$P(A_{k})=P(\omega\in\mathcal{S}^{D,k}|\hat{\mathbf{X}}_{T}^{k})$. Then the
non-identifiability in the multivariate time series setting can be rewritten
as:

$\displaystyle
P(\omega\in\\{\mathcal{S}^{D,k}\\}_{k=1}^{K}|\hat{\mathbf{X}}_{T})$
$\displaystyle=$ $\displaystyle P(A_{1},A_{2},...,A_{K})$ $\displaystyle=$
$\displaystyle P(A_{1})P(A_{2}|A_{1})P(A_{3}|A_{2},A_{1})\cdots
P(A_{K}|A_{K-1},...,A_{1})$ $\displaystyle=$ $\displaystyle
P(\omega\in\mathcal{S}^{D,1}|\hat{\mathbf{X}}_{T}^{1})\cdot\beta_{2}\cdot\beta_{3}\cdots\beta_{K}$
(.1) $\displaystyle\leq$ $\displaystyle
P(\omega\in\mathcal{S}^{D,1}|\hat{\mathbf{X}}_{T}^{1})\cdot(\max_{i\in\\{2,...,K\\}}\beta_{i})^{K-1}$
(.2)

In Eq. (.1), $\beta_{i},i=2,...,K$ denotes the probability that any parameter
$\omega\in\\{\mathcal{S}^{D,k}\\}_{k=1}^{i-1}$ falls into $\mathcal{S}^{D,i}$,
i.e., $P(A_{i}|A_{i-1},...,A_{1})$. In other words, $\beta_{i}$ means the
overlapping ratio between $\mathcal{S}^{D,i}$ and
$\cap_{k=1}^{i-1}\mathcal{S}^{D,k}$, and $\beta_{i}\leq 1$. Especially,
$\beta_{i}=1$ only if
$\cap_{k=1}^{i-1}\mathcal{S}^{D,k}\subseteq\mathcal{S}^{D,i}$, indicating that
the $i$-th feature is fully dependent on the first $i-1$ features and can be
avoided with properly chosen features. Also note that the order of features in
Eq. (.1) is arbitrary. Eq. (.2) defines the upper bound of the non-
identifiability in the multivariate time series setting, the equality holds if
$\beta_{i}=\beta_{j},i=2,...,K,j=2,...,K,i\neq j$ or
$\mathcal{S}^{D,k}=\Omega^{D,k},k=2,...,K$, which can also be avoided with
proper choice of the features and $D$. Therefore, it is clear that the non-
identifiability of calibrating $K$-variate time series decreases exponentially
with the number of features $K$, as it is highly likely
$(\max_{i=2,...,K}\beta_{i})<1$.

###  3.3 Aggregating Multiple Features via Maximization

Note that, the above discussions require the multivariate time series
$\\{\hat{\mathbf{X}}_{T}^{k}\\}_{k=1}^{K}$ being calibrated separately as $K$
univariate time series with existing discrepancy metrics $D$. This results in
$K$ discrepancy values of
$D(\hat{\mathbf{X}}_{T}^{k},M(w)=\mathbf{X}_{T}^{k}),k=1,\cdots,K$. This
section discusses how to optimally aggregate the $K$ values, so that the
practical alleviation of the non-identifiability follows the above analysis.

Let $F$ denote the proposed objective function for calibrating the
multivariate $\hat{\mathbf{X}}_{T}$. To achieve
$\mathcal{S}^{F}=\cap_{k=1}^{K}\mathcal{S}^{D,k}$, the proposed way is to
ensure every location in $\Omega^{F}$ to be the worst among all $K$ parameter
space $\\{\Omega^{D,k}\\}_{k=1}^{K}$ (see Figure 3 for illustration). In this
case, any parameter $\omega$ that is not simultaneously in all $K$ non-
identifiable sets will make it fall outside of the $\mathcal{S}^{F}$. This is
because that the parameters outside the non-identifiable set is worst than
those who are non-identifiable from the ground-truth parameter $\omega^{*}$.

Given that the calibration problem is a minimization problem, the new
objective function $F$ can be implemented with a maximization aggregation
among $K$ individual calibration tasks with $D$. That is,

$\displaystyle\min_{\omega\in\Omega}F(\hat{\mathbf{X}}_{T},M(w)=\mathbf{X}_{T})$
(6) $\displaystyle=\min_{\omega\in\Omega}$
$\displaystyle\max_{k\in\\{1,\cdots,K\\}}D(\hat{\mathbf{X}}_{T}^{k},M(w)=\mathbf{X}_{T}^{k})$

Figure 3: Optimizing the maximization of two functions is equivalent to the
search in the intersection between the two search spaces.

##  4 Empirical Studies

This section empirically verifies two major claims of this paper.

  * •

With the increasing number of features applied in the calibration, the non-
identifiable set shrinks largely.

  * •

The fidelity of the simulation data, in terms of various performance metrics,
is improved significantly by calibrating more features.

###  4.1 Experimental Settings

To conduct a calibration process, 4 terms should be specified according to Eq.
(6), i.e., the discrepancy metric $D$, the simulation model $M(\omega)$, the
optimization algorithm $\rm{min}_{\omega}$, and the $K$ features
$\\{\hat{\mathbf{X}}_{T}^{k}\\}_{k=1}^{K}$.

The discrepancy metric. The Wasserstein distance is employed as $D$ since it
provides geometric interpretability, and tail sensitivity [25, 26]. Given two
univariate time series $\mathbf{X}_{T}^{k}$ and $\hat{\mathbf{X}}_{T}^{k}$,
the Wasserstein distance is defined as

$D(\hat{\mathbf{X}}_{T}^{k},\mathbf{X}_{T}^{k})=\int_{-\infty}^{+\infty}|U(x)-V(x)|\rm{d}\textit{x}$
(7)

where

$\displaystyle
U(x)=\frac{1}{T}\sum_{t=t_{s}}^{t_{e}}\mathcal{I}(\hat{\mathbf{X}}_{t}^{k}\leq
x)$ (8) $\displaystyle
V(x)=\frac{1}{T}\sum_{t=t_{s}}^{t_{e}}\mathcal{I}(\mathbf{X}_{t}^{k}\leq x)$

Here $U$ and $V$ are the empirical cumulative distribution functions for the
data set $\hat{\mathbf{X}}_{T}^{k}$ and $\mathbf{X}_{T}^{k}$, respectively.
And $\mathcal{I}(\cdot)$ is an indicator function that returns 1 if the input
event is true, otherwise it returns 0.

The simulation model. This work considers the Preis-Golke-Paul-Schneid (PGPS)
model as the simulation model, due to its popularity in recent financial
market simulation works [11, 6]. The PGPS model simulates the market dynamics
through interactions between two types of agents, i.e., 125 liquidity
providers and 125 liquidity takers. At the $t$-th time-step, each liquidity
provider submits a limited order at a fixed probability $\alpha$ with the
default volume equals 1. The probability of a limited order being bid side or
ask side equals to 0.5. Each liquidity taker submits a market order at a fixed
probability $\mu$ with the default volume equals 1. The probability of a
market order being bid side or ask side is $q_{taker}(t)$ or $1-q_{taker}(t)$,
respectively. The probability $q_{taker}(t)$ is specified by a mean-reverting
random walk with mean equals 0.5, mean reversion probability equals
$0.5+|q_{taker}(t)-0.5|$, and the increment sizes towards mean equals
$\pm\Delta_{s}$. Moreover, the liquidity taker has a probability $\delta$ to
cancel its untraded limited order. Let $p_{a}(t)$ and $p_{b}(t)$ represents
the best ask and bid price, respectively. The price of a market order is
determined by the market, i.e., the best price at the opposite side of the
LOB. The price of a limited order is determined as
$p_{a}(t)+(\lambda(t)\log(u))+s$, where $s=1$ if it is a ask order and
otherwise $s=-1$, and $u\sim U(0,1)$. Here $\lambda(t)$ is a time-variant
order placement depth parameter and is calculated as

$\lambda(t)=\lambda_{0}(1+\frac{|q_{taker}(t)-\frac{1}{2}|}{\sqrt{<q_{taker}(t)-\frac{1}{2}>^{2}}}C_{\lambda})$
(9)

$\sqrt{<q_{taker}(t)-\frac{1}{2}>^{2}}$ is a pre-computed value. Before
initiating the simulation, it is computed in advance through $10^{5}$ Monte
Carlo iterations, ensuring its convergence to the accurate standard deviation.

To summarize, the PGPS model has 6 hyper-parameters shared by all agents that
need to be calibrated, i.e.,
$\omega=[\delta,\lambda_{0},C_{\lambda},\Delta_{s},\alpha,\mu]$.

The optimization method. Since the order matching process needs to align with
the real market, the PGPS model should be run on the executable programs based
some trading simulators222This work builds the simulation environment based on
the Multi-Agent Exchange Environment (MAXE) simulator developed by University
of Oxford [27].. As a result, the calibration objective function is non-
differentiable and the calibration process often has to resort to non-
derivative optimization methods. In such contexts, both Bayesian optimization
methods and evolutionary algorithms are particularly suitable. This work
employs the well-known Particle Swarm Optimization (PSO) algorithm [28] to
optimize $\omega$ with respect to Eq.(6). For simplicity, the standard version
of PSO [29] is considered. Without further fine-tuning, the hyper-parameters
of PSO follow the suggested configurations, where the population size is set
to 40, the inertia weight is set to 0.8, the cognitive and social crossover
parameters $c_{1}=0.5,c_{2}=0.5$.

The selected features. Six different yet commonly seen features are considered
[15, 30], which mainly concern either the price or volume information of the
LOB data and are listed as follows:

  * •

$f_{1}$, the mid-price $m_{t}=\frac{p_{a}(t)+p_{b}(t)}{2}$ at each $t$-th
step;

  * •

$f_{2}$, the total traded volume within each $t$-th step;

  * •

$f_{3}$, the mid-price return within each $t$-th step $\ln m_{t+1}-\ln m_{t}$;

  * •

$f_{4}$, the spread at each $t$-th step $p_{a}(t)-p_{b}(t)$;

  * •

$f_{5}$, the total volume of the best bid price at each $t$-th step;

  * •

$f_{6}$, the total volume of the best ask price at each $t$-th step.

By incorporating the 6 selected features, 6 objective functions are
constructed based on Eq.(6), denoted as $F_{i},i=1,2,...,6$. Each $F_{i}$
indicates that the first $i$ features are used for calibration, i.e.,
$F_{i}=\max_{k\in\\{1,\cdots,i\\}}D(\hat{\mathbf{X}}_{T}^{k},M(\omega))$. In
the following empirical studies, it is shown how the increasing number of
features influences the simulation fidelity.

Parameters | $\mathbf{\delta}$ | $\mathbf{\lambda_{0}}$ | $\mathbf{C_{\lambda}}$ | $\mathbf{\Delta_{S}}$ | $\mathbf{\alpha}$ | $\mathbf{\mu}$
---|---|---|---|---|---|---
data 1 | 0.025 | 100 | 10 | 0.001 | 0.15 | 0.025
data 2 | 0.002 | 200 | 10 | 0.002 | 0.1 | 0.03
data 3 | 0.05 | 152 | 45 | 0.003 | 0.13 | 0.05
data 4 | 0.02 | 288 | 27 | 0.003 | 0.11 | 0.04
data 5 | 0.04 | 62 | 35 | 0.003 | 0.12 | 0.04
data 6 | 0.01 | 171 | 15 | 0.0015 | 0.15 | 0.03
data 7 | 0.033 | 114 | 14 | 0.0033 | 0.1 | 0.047
data 8 | 0.01 | 129 | 30 | 0.0017 | 0.05 | 0.03
data 9 | 0.02 | 62 | 2 | 0.003 | 0.14 | 0.02
data 10 | 0.01 | 87 | 23 | 0.001 | 0.09 | 0.05
Table 1: Parameter settings of generating 10 synthetic data with the PGPS
model.

The test protocol. The test data contains 11 targeted time series to be
calibrated, including 10 synthetic data and 1 real data. The 10 synthetic time
series are generated by setting 10 different parameters of the PGPS model as
shown in Table 1, each of which consists of 3600 time steps at the frequency
of 1 second. The real data is chosen as 000001.sz from Shenzhen Stock Exchange
of China, consisting of 1200 time steps at a frequency of 3 seconds from 9:30
a.m. to 10:30 a.m. of a day in 2019. The time budget of $10^{4}$ simulation
evaluations is allowed for the calibration of each targeted data. That is, the
PSO runs for 250 iterations with population size 40, and the best parameter
with the lowest objective function value is applied into PGPS model to
simulate data as $\mathbf{X}_{T}$. The performance of the calibration is
quantitatively validated using the Wasserstein (W) distance and the Mean
Square Error (MSE) commonly seen in time series analysis. Both metrics are
applied to measure the discrepancy between the synthetic data and simulated
data on each feature, normalized with the min-max method, respectively.

(a) $F_{1}$

(b) $F_{2}$

(c) $F_{3}$

(d) $F_{4}$

(e) $F_{5}$

(f) $F_{6}$

Figure 4: With the increasing number of features, the cardinality of the non-
identifiable set decreases significantly.

###  4.2 Results and Analysis

Effectiveness of multiple features. To show how the increasing numbers of
features alleviate the non-identifiability issue, we depict the changes of the
2-dimensional parameter space of $[\alpha,\mu]$. Specifically, a target data
of 3600 time steps is generated using the PGPS model with the commonly
recommended default parameter setting
$\omega^{*}=[0.025,100,10,0.001,0.15,0.025]$ [6]. On this default setting,
10000 combinations are created by sampling in $[\alpha,\mu]$ in grid. By
calculating the objective value $F$ between the target data and each of the
10000 combinations, it obtains the objective values of the 10000 combinations
in the space of $[\alpha,\mu]$, which are depicted as Figure 4.(a)-(f). The 6
figures in Figure 4.(a)-(f) are depicted according to the objective values of
$F_{1}$, $F_{2}$,…,$F_{6}$, respectively. Only the combinations whose
objective values rank in the best 10% over all the corresponding features, are
they depicted in blue, otherwise they will be depicted in yellow. For example,
in Figure 4.(c), the combinations depicted in blue are those whose discrepancy
values rank in 10% among the 10000 combinations on all the first 3 features
$\\{f_{i}\\}_{i=1}^{3}$. By this means, the blue points indicate the
intersection of the combinations who rank best 10% in all the corresponding
features in $F$. In this regard, the non-identifiable set are also depicted in
blue as they are indistinguishable with $\omega^{*}$.

As can be seen that, with the increaseha of the features, the blue set, as
well as the non-identifiable set, decreases significantly. When 5 to 6
features are included in the objective function, there leaves very few blue
points, indicating that the non-identifiability issue is indeed alleviated
very effectively.

Note that, although the non-identifiable set of any $F_{i}$ is verified to be
smaller than that of any $F_{j}$, $i>j$, it does not deterministically lead to
better calibration performance of $F_{i}$ than $F_{j}$. Because $F$ mainly
decides the parameter space, while the calibration performance is also jointly
influenced by the optimization algorithms. Given the calibration problem is
black-box with multiple local optima, it is non-trivial for the optimization
algorithms to find the global optimum even the parameter space is modeled
fully identifiable.

data $obj$ | $F_{1}$ | $F_{2}$ | $F_{3}$ | $F_{4}$ | $F_{5}$ | $F_{6}$
---|---|---|---|---|---|---
Data 1 | W | 7.49 | 3.17 | 4.85 | 6.25 | 2.59 | 4.90
MSE | 6.97 | 4.46 | 3.93 | 6.25 | 4.38 | 4.04
Data 2 | W | 3.68 | 1.58 | 2.60 | 5.81 | 4.40 | 3.88
MSE | 4.15 | 4.18 | 4.88 | 5.25 | 5.15 | 4.33
Data 3 | W | 2.34 | 5.52 | 6.74 | 4.79 | 5.95 | 8.07
MSE | 5.03 | 4.00 | 4.42 | 3.62 | 3.58 | 4.38
Data 4 | W | 1.17 | 1.99 | 1.72 | 4.54 | 1.98 | 5.70
MSE | 3.26 | 3.04 | 3.65 | 3.19 | 3.36 | 3.90
Data 5 | W | 3.97 | 3.66 | 3.03 | 7.88 | 7.65 | 2.54
MSE | 4.85 | 3.56 | 3.65 | 6.08 | 4.01 | 3.66
Data 6 | W | 6.15 | 2.52 | 7.34 | 5.98 | 2.42 | 7.45
MSE | 5.65 | 4.4 | 5.62 | 5.87 | 4.54 | 6.44
Data 7 | W | 5.66 | 5.10 | 3.29 | 2.38 | 0.99 | 1.99
MSE | 5.20 | 3.76 | 3.88 | 3.71 | 3.55 | 3.72
Data 8 | W | 2.64 | 0.88 | 2.32 | 3.88 | 3.96 | 2.23
MSE | 3.11 | 2.78 | 3.48 | 3.90 | 4.12 | 2.81
Data 9 | W | 9.28 | 1.88 | 3.10 | 8.74 | 8.71 | 4.42
MSE | 4.08 | 3.86 | 7.43 | 7.65 | 5.02 | 3.71
Data 10 | W | 1.25 | 5.40 | 2.12 | 0.99 | 0.57 | 0.90
MSE | 2.88 | 3.68 | 2.58 | 2.64 | 2.65 | 2.62
#win | W | 0 | 7 | 6 | 5 | 5 | 6
MSE | 0 | 8 | 6 | 5 | 6 | 7
Table 2: Calibrating with the multivariate time series ($F_{2}$-$F_{6}$)
robustly outperforms that with the univariate data ($F_{1}$).

Calibration of the synthetic data. For each of the 10 synthetic data, by
optimizing the 6 objective functions $F_{i},i=1,...,6$ with PSO, it results in
6 calibrated models and 6 simulated data. Note that, even though a model is
calibrated using $F_{i},i<6$, it can still functionally generate the simulated
data $\mathbf{X}_{T}$ with respect to all 6 features.

Figure 5: The simulated data using the proposed method resembles the synthetic
data on all 6 features.

All 6 discrepancy values $D$ between each simulated data and its corresponding
synthetic data is measured by both the W and MSE, based on the 6 features,
respectively. In Table 2, each cell displays the average of these 6
discrepancy values. It shows that by using more than one feature ($F_{2}$ to
$F_{6}$) for calibration, it consistently outperforms the calibration with the
univariate data ($F_{1}$). Specifically, on the synthetic data of data 1, 3,
and 7, calibrating with $F_{1}$ is defeated by all of the other 5 objective
functions who use at least two features. On other instances, there are at
least two objectives among $F_{2}$ to $F_{6}$ outperform $F_{1}$.

To a further step, the numbers of synthetic data that each $F_{i},i=2,...,6$
outperforms $F_{1}$ are counted and recorded in the last row of Table 2. It
clearly shows that each $F_{i},i=2,...,6$ wins $F_{1}$ on at least half of the
testing instances on both W and MSE metrics. More specifically, $F_{2}$ wins
the most among $F_{2}$ to $F_{6}$. Note that, all these 5 objectives employ
the second feature of total traded volume. It can be attributed that this
feature contributes the most to the calibration of multiple features. This is
reasonable as the traded volume is a data feature very different to the first
feature of mid-price.

Figure 6: By calibrating with more features, the simulated data resembles the
real market data better.

Then, we depict the high-frequency time series of 6 features to show how the
simulated data resembles the synthetic data in Figure 5. Specifically, the
simulated data calibrated by $F_{5}$ on data 7 is displayed as it represents
the best calibration performance that $F_{2}$ to $F_{6}$ all outperform
$F_{1}$ on both the Wasserstein distance and MSE metric. It can be seen that
the simulated time series resemble the synthetic data on all 6 features very
closely. To our best knowledge, this is the first ABM-agnostic method of
successfully calibrating the high-frequency market time series at 1 second
level and simulating 6 features with high-fidelity. In comparison, existing
works mostly calibrate the mid-price data at a daily frequency [6].

Calibration of the real data. At last, it is verified that the proposed method
can also calibrate the high-frequency real market data at a 3-second level. A
1-hour time series of 000001.sz is calibrated using $F_{1}$, $F_{2}$, and
$F_{3}$. The simulated data generated by the corresponding best calibrated
models is depicted in Figure 6, together with the real observed data. Due to
the limited space, only the mid-price time series is shown in a time step of 1
minute. It can be seen that the simulated mid-price of $F_{1}$ is less similar
to the real observed data. By incorporating more features, the fidelity of the
simulated mid-price improves by $F_{2}$ and $F_{3}$. This confirms that the
proposed method indeed can calibrate the real market data with multiple time
series, thus suggesting a new effective calibration objective function for
financial market simulation.

##  5 Conclusions

This paper studies the non-identifiability issue of social simulation. To our
best knowledge, this is the first time that the non-identifiability is
formally defined and analyzed to be alleviated. It is defined as the
probability of randomly sampling in the non-identifiable set of the parameter
space. It is rigorously analyzed that this probability can be reduced with
more time series features in an exponential manner. Based on the analysis, a
new objective function is proposed to effectively aggregate multiple features
with a maximization function over existing discrepancy metrics. Extensive
empirical studies have been conducted on 10 synthetic data and 1 real data of
the financial market simulation tasks. It has been successfully verified in
the simulations that, with multiple features, not only the non-identifiable
set can be minimized but the simulation fidelity is largely improved. Hence,
this paper provides a high-fidelity objective function to calibrate the high-
frequency financial market simulator.

## References

  * [1] Flaminio Squazzoni, Wander Jager, and Bruce Edmonds.  Social simulation in the social sciences: A brief overview.  Social Science Computer Review, 32(3):279–294, 2014.
  * [2] J Doyne Farmer and Duncan Foley.  The economy needs agent-based modelling.  Nature, 460(7256):685–686, 2009.
  * [3] Annalisa Fabretti.  On the problem of calibrating an agent based model for financial markets.  Journal of Economic Interaction and Coordination, 8:277–293, 2013.
  * [4] Dongjun Kim, Tae-Sub Yun, Il-Chul Moon, and Jang Won Bae.  Automatic calibration of dynamic and heterogeneous parameters in agent-based models.  Autonomous Agents and Multi-Agent Systems, 35(2):46, 2021.
  * [5] Donovan Platt.  A comparison of economic agent-based model calibration methods.  Journal of Economic Dynamics and Control, 113:103859, 2020.
  * [6] Kelly Goosen.  Calibrating high frequency trading data to agent based models using approximate bayesian computation.  Master’s thesis, Faculty of Science, 2021.
  * [7] Yuanlu Bai, Henry Lam, Tucker Balch, and Svitlana Vyetrenko.  Efficient calibration of multi-agent simulation models from output series with bayesian optimization.  In Proceedings of the Third ACM International Conference on AI in Finance, pages 437–445, 2022.
  * [8] Richard G Palmer, W Brian Arthur, John H Holland, Blake LeBaron, and Paul Tayler.  Artificial economic life: a simple model of a stockmarket.  Physica D: Nonlinear Phenomena, 75(1-3):264–274, 1994.
  * [9] Adamantios Ntakaris, Martin Magris, Juho Kanniainen, Moncef Gabbouj, and Alexandros Iosifidis.  Benchmark dataset for mid-price forecasting of limit order book data with machine learning methods.  Journal of Forecasting, 37(8):852–866, 2018.
  * [10] Giorgio Fagiolo, Alessio Moneta, and Paul Windrum.  A critical guide to empirical validation of agent-based models in economics: Methodologies, procedures, and open problems.  Computational Economics, 30:195–226, 2007.
  * [11] Donovan Platt and Tim Gebbie.  Can agent-based models probe market microstructure?  Physica A: Statistical Mechanics and its Applications, 503:1092–1106, 2018.
  * [12] James Paulin, Anisoara Calinescu, and Michael Wooldridge.  Agent-based modeling for complex financial systems.  IEEE Intelligent Systems, 33(2):74–82, 2018.
  * [13] Shu-Heng Chen, Chia-Ling Chang, and Ye-Rong Du.  Agent-based economic models and econometrics.  The Knowledge Engineering Review, 27(2):187–219, 2012.
  * [14] Tobias Preis, Sebastian Golke, Wolfgang Paul, and Johannes J Schneider.  Multi-agent-based order book model of financial markets.  Europhysics Letters, 75(3):510, 2006.
  * [15] Svitlana Vyetrenko, David Byrd, Nick Petosa, Mahmoud Mahfouz, Danial Dervovic, Manuela Veloso, and Tucker Balch.  Get real: Realism metrics for robust limit order book market simulations.  In Proceedings of the First ACM International Conference on AI in Finance, pages 1–8, 2020.
  * [16] Carl Chiarella and Giulia Iori.  A simulation analysis of the microstructure of double auction markets.  Quantitative finance, 2(5):346, 2002.
  * [17] James Paulin, Anisoara Calinescu, and Michael Wooldridge.  Understanding flash crash contagion and systemic risk: A micro–macro agent-based approach.  Journal of Economic Dynamics and Control, 100:200–229, 2019.
  * [18] Eric Weinstein.  Meltdown modelling.  Nature, 460:6, 2009.
  * [19] Jakob Grazzini and Matteo Richiardi.  Estimation of ergodic agent-based models by simulated minimum distance.  Journal of Economic Dynamics and Control, 51:148–165, 2015.
  * [20] Francesco Lamperti.  An information theoretic criterion for empirical validation of simulation models.  Econometrics and Statistics, 5:83–106, 2018.
  * [21] Sylvain Barde.  A practical, accurate, information criterion for nth order markov processes.  Computational Economics, 50(2):281–324, 2017.
  * [22] Jiri Kukacka and Jozef Barunik.  Estimation of financial agent-based models with simulated maximum likelihood.  Journal of Economic Dynamics and Control, 85:21–45, 2017.
  * [23] Jakob Grazzini, Matteo G Richiardi, and Mike Tsionas.  Bayesian estimation of agent-based models.  Journal of Economic Dynamics and Control, 77:26–47, 2017.
  * [24] Francesco Lamperti, Andrea Roventini, and Amir Sani.  Agent-based model calibration using machine learning surrogates.  Journal of Economic Dynamics and Control, 90:366–389, 2018.
  * [25] Andrea Coletta, Aymeric Moulin, Svitlana Vyetrenko, and Tucker Balch.  Learning to simulate realistic limit order book markets from data as a world agent.  In Proceedings of the Third ACM International Conference on AI in Finance, pages 428–436, 2022.
  * [26] Andrea Coletta, Matteo Prata, Michele Conti, Emanuele Mercanti, Novella Bartolini, Aymeric Moulin, Svitlana Vyetrenko, and Tucker Balch.  Towards realistic market simulations: a generative adversarial networks approach.  In Proceedings of the Second ACM International Conference on AI in Finance, pages 1–9, 2021.
  * [27] Peter Belcak, Jan-Peter Calliess, and Stefan Zohren.  Fast agent-based simulation framework with applications to reinforcement learning and the study of trading latency effects.  In International Workshop on Multi-Agent Systems and Agent-Based Simulation, pages 42–56. Springer, 2021.
  * [28] James Kennedy and Russell Eberhart.  Particle swarm optimization.  In Proceedings of ICNN’95-international conference on neural networks, volume 4, pages 1942–1948. IEEE, 1995.
  * [29] Yuhui Shi and Russell Eberhart.  A modified particle swarm optimizer.  In Proceedings of 1998 IEEE international conference on evolutionary computation, pages 69–73. IEEE, 1998.
  * [30] Rama Cont.  Empirical properties of asset returns: stylized facts and statistical issues.  Quantitative finance, 1(2):223, 2001.